2021-02-12T17:46:34.546 INFO:root:teuthology version: 1.1.0-87d40f79 2021-02-12T17:46:34.547 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --name sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi --owner scheduled_sseshasa@teuthology --archive /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860 --description rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} -- /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860/orig.config.yaml 2021-02-12T17:46:34.580 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2021-02-12T17:46:34.633 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860 branch: wip-dmclock-fix-high-client-profile description: rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}} email: null first_in_suite: false job_id: '5877860' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi name: sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi nuke-on-error: true openstack: - volumes: count: 3 size: 10 os_type: centos os_version: '8.2' overrides: admin_socket: branch: wip-dmclock-fix-high-client-profile ceph: conf: global: mon client directed command retry: 5 mon client hunt interval max multiple: 2 mon election default strategy: 3 mon mgr beacon grace: 90 ms bind msgr2: false ms inject socket failures: 1000 ms type: async mgr: debug mgr: 20 debug monc: 10 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon scrub interval: 300 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 20 debug bluestore: 20 debug filestore: 20 debug journal: 20 debug ms: 20 debug osd: 25 debug rocksdb: 10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd debug verify cached snaps: true osd debug verify missing on start: true osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore osd op queue: debug_random osd op queue cut off: debug_random fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME log-whitelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) mon_bind_msgr2: false sha1: 5a3430b93fcfcd31d38534758b8997c894688c4b ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 install: ceph: sha1: 5a3430b93fcfcd31d38534758b8997c894688c4b selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-dmclock-fix-high-client-profile sha1: 5a3430b93fcfcd31d38534758b8997c894688c4b owner: scheduled_sseshasa@teuthology priority: 70 repo: git://git.ceph.com/ceph-ci.git roles: - - mon.a - mon.b - mon.c - mgr.x - osd.0 - osd.1 - osd.2 sha1: 5a3430b93fcfcd31d38534758b8997c894688c4b sleep_before_teardown: 0 suite: rados suite_branch: wip-dmclock-fix-high-client-profile suite_path: /home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa suite_relpath: qa suite_repo: git://git.ceph.com/ceph-ci.git suite_sha1: 5a3430b93fcfcd31d38534758b8997c894688c4b targets: smithi107.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCU7A7ySnMnqXkw6mW2Ut7f3wWgG3dEhpElLMt5wQt49bDKJZh1Geh7/fnEHP6nS1p/cc6LPv9crBJShjp2APgkTMzAr2xPb30BJsOV6MpevHZRwE3tgfGXAp6Hyqzz5+iMbbnvb1oZRp/mYhO+aNuFH++RcohkB0LDBVTC1kz+yfIzFJk7P//V53ykuf77O/8LNBfh0ZaJ8reur1K1ppwfU6fOG9wbcL3GZRraETHDm1lFabdMLpqiU5roPqKff4IYdZnCFyvgRTlRsH9AjgEWsvSWs6IHsoswihwQsSXZy416u6twSN56MFc2yArCaGKH6B1f7okbvXbmx2Kzhh5U6Fssu/ajVjl663/L88eXDr5pEIbjXCkAGAV8/PID+/xe7qqH4fKAd2XojJ8mgGKdk2DOR8DvQDlekCzlIeqD0JWKS3xeylAtXoA/ypFZIuxYspEJ/WWAwjSyWm/6LyIn+6XjYgD8+pzkpNa2w6bjbBos0r4dOtctSfhg4UyerPk= tasks: - install: null - ceph: conf: osd: osd min pg log entries: 5 log-ignorelist: - but it is still running - overall HEALTH_ - \(OSDMAP_FLAGS\) - \(OSD_ - \(PG_ - \(OBJECT_ pre-mgr-commands: - sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force - osd_backfill: null teuthology_branch: master teuthology_sha1: 87d40f79d7da6caf6da06c190938773bf745140c timestamp: 2021-02-12_17:37:51 tube: smithi user: sseshasa verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.29888 2021-02-12T17:46:34.633 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa; will attempt to use it 2021-02-12T17:46:34.634 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa/tasks 2021-02-12T17:46:34.634 INFO:teuthology.run_tasks:Running task internal.check_packages... 2021-02-12T17:46:34.636 INFO:teuthology.task.internal:Checking packages... 2021-02-12T17:46:34.655 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'basic' and ceph hash '5a3430b93fcfcd31d38534758b8997c894688c4b' 2021-02-12T17:46:34.655 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2021-02-12T17:46:34.656 INFO:teuthology.packaging:ref: None 2021-02-12T17:46:34.656 INFO:teuthology.packaging:tag: None 2021-02-12T17:46:34.656 INFO:teuthology.packaging:branch: wip-dmclock-fix-high-client-profile 2021-02-12T17:46:34.656 INFO:teuthology.packaging:sha1: 5a3430b93fcfcd31d38534758b8997c894688c4b 2021-02-12T17:46:34.656 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-dmclock-fix-high-client-profile 2021-02-12T17:46:34.818 INFO:teuthology.task.internal:Found packages for ceph version 17.0.0-713.g5a3430b9 2021-02-12T17:46:34.823 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2021-02-12T17:46:34.830 INFO:teuthology.task.internal:no buildpackages task found 2021-02-12T17:46:34.830 INFO:teuthology.run_tasks:Running task internal.save_config... 2021-02-12T17:46:34.985 INFO:teuthology.task.internal:Saving configuration 2021-02-12T17:46:34.997 INFO:teuthology.run_tasks:Running task internal.check_lock... 2021-02-12T17:46:35.007 INFO:teuthology.task.internal.check_lock:Checking locks... 2021-02-12T17:46:35.025 DEBUG:teuthology.task.internal.check_lock:machine status is {'is_vm': False, 'locked': True, 'locked_since': '2021-02-12 17:41:29.932472', 'description': '/home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860', 'locked_by': 'scheduled_sseshasa@teuthology', 'up': True, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBLHYjKqVIzKSCiSe2e119NJHt/7NXR0MC0X95YPyPV4pHnKTOBvKy8oTuCeKOYmMnOUlEmrClQpXkKKxD/dIrwQ=', 'os_version': '8', 'machine_type': 'smithi', 'vm_host': None, 'os_type': 'centos', 'arch': 'x86_64', 'mac_address': None, 'name': 'smithi107.front.sepia.ceph.com'} 2021-02-12T17:46:35.025 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2021-02-12T17:46:35.034 INFO:teuthology.task.internal:roles: ubuntu@smithi107.front.sepia.ceph.com - ['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2'] 2021-02-12T17:46:35.034 INFO:teuthology.run_tasks:Running task console_log... 2021-02-12T17:46:35.071 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f6c5c18b158>, signals=[15]) 2021-02-12T17:46:35.071 INFO:teuthology.run_tasks:Running task internal.connect... 2021-02-12T17:46:35.081 INFO:teuthology.task.internal:Opening connections... 2021-02-12T17:46:35.081 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:46:35.082 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:46:35.153 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2021-02-12T17:46:35.158 DEBUG:teuthology.orchestra.run.smithi107:> uname -m 2021-02-12T17:46:35.305 INFO:teuthology.orchestra.run.smithi107.stdout:x86_64 2021-02-12T17:46:35.305 DEBUG:teuthology.orchestra.run.smithi107:> cat /etc/os-release 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:NAME="CentOS Linux" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION="8 (Core)" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:ID="centos" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:ID_LIKE="rhel fedora" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:VERSION_ID="8" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:PLATFORM_ID="platform:el8" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:PRETTY_NAME="CentOS Linux 8 (Core)" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:ANSI_COLOR="0;31" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:HOME_URL="https://www.centos.org/" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout:BUG_REPORT_URL="https://bugs.centos.org/" 2021-02-12T17:46:35.359 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:46:35.360 INFO:teuthology.orchestra.run.smithi107.stdout:CENTOS_MANTISBT_PROJECT="CentOS-8" 2021-02-12T17:46:35.360 INFO:teuthology.orchestra.run.smithi107.stdout:CENTOS_MANTISBT_PROJECT_VERSION="8" 2021-02-12T17:46:35.360 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT="centos" 2021-02-12T17:46:35.360 INFO:teuthology.orchestra.run.smithi107.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="8" 2021-02-12T17:46:35.360 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:46:35.360 INFO:teuthology.lock.ops:Updating smithi107.front.sepia.ceph.com on lock server 2021-02-12T17:46:35.375 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2021-02-12T17:46:35.385 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2021-02-12T17:46:35.413 INFO:teuthology.task.internal:Checking for old test directory... 2021-02-12T17:46:35.413 DEBUG:teuthology.orchestra.run.smithi107:> test '!' -e /home/ubuntu/cephtest 2021-02-12T17:46:35.426 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2021-02-12T17:46:35.434 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2021-02-12T17:46:35.434 DEBUG:teuthology.orchestra.run.smithi107:> test -z $(ls -A /var/lib/ceph) 2021-02-12T17:46:35.496 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2021-02-12T17:46:35.522 INFO:teuthology.run_tasks:Running task kernel... 2021-02-12T17:46:35.534 DEBUG:teuthology.task.kernel:normalized config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'mon.c': {'kdb': True, 'sha1': 'distro'}, 'mgr.x': {'kdb': True, 'sha1': 'distro'}, 'osd.0': {'kdb': True, 'sha1': 'distro'}, 'osd.1': {'kdb': True, 'sha1': 'distro'}, 'osd.2': {'kdb': True, 'sha1': 'distro'}} 2021-02-12T17:46:35.534 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2021-02-12T17:46:35.534 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2021-02-12T17:46:35.549 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-193.28.1.el8_2.x86_64 2021-02-12T17:46:35.549 INFO:teuthology.task.kernel:Running kernel on smithi107: 4.18.0-193.28.1.el8_2.x86_64 2021-02-12T17:46:35.550 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum install -y kernel 2021-02-12T17:46:38.059 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - AppStream 36 MB/s | 6.3 MB 00:00 2021-02-12T17:46:39.142 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - Base 24 MB/s | 2.3 MB 00:00 2021-02-12T17:46:39.597 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - Extras 134 kB/s | 8.6 kB 00:00 2021-02-12T17:46:40.065 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 21 MB/s | 8.9 MB 00:00 2021-02-12T17:46:41.947 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 4.9 MB/s | 24 kB 00:00 2021-02-12T17:46:43.012 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-193.el8.x86_64 is already installed. 2021-02-12T17:46:43.013 INFO:teuthology.orchestra.run.smithi107.stdout:Package kernel-4.18.0-193.28.1.el8_2.x86_64 is already installed. 2021-02-12T17:46:43.206 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:46:43.215 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:46:43.215 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:46:43.215 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: kernel x86_64 4.18.0-240.10.1.el8_3 CentOS-Base 4.3 M 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: linux-firmware noarch 20200619-101.git3890db36.el8_3 CentOS-Base 101 M 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core x86_64 4.18.0-240.10.1.el8_3 CentOS-Base 30 M 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules x86_64 4.18.0-240.10.1.el8_3 CentOS-Base 26 M 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout:Install 3 Packages 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 1 Package 2021-02-12T17:46:43.216 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:46:43.217 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 162 M 2021-02-12T17:46:43.217 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:46:43.311 INFO:teuthology.orchestra.run.smithi107.stdout:(1/4): kernel-4.18.0-240.10.1.el8_3.x86_64.rpm 46 MB/s | 4.3 MB 00:00 2021-02-12T17:46:44.836 INFO:teuthology.orchestra.run.smithi107.stdout:(2/4): kernel-core-4.18.0-240.10.1.el8_3.x86_64 19 MB/s | 30 MB 00:01 2021-02-12T17:46:45.015 INFO:teuthology.orchestra.run.smithi107.stdout:(3/4): kernel-modules-4.18.0-240.10.1.el8_3.x86 14 MB/s | 26 MB 00:01 2021-02-12T17:46:45.844 INFO:teuthology.orchestra.run.smithi107.stdout:(4/4): linux-firmware-20200619-101.git3890db36. 40 MB/s | 101 MB 00:02 2021-02-12T17:46:45.847 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:46:45.847 INFO:teuthology.orchestra.run.smithi107.stdout:Total 61 MB/s | 162 MB 00:02 2021-02-12T17:46:45.937 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:46:46.223 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:46:46.223 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:46:47.887 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:46:47.888 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:46:57.843 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:46:59.603 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : linux-firmware-20200619-101.git3890db36.el8_3.noarch 1/5 2021-02-12T17:47:00.131 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-core-4.18.0-240.10.1.el8_3.x86_64 2/5 2021-02-12T17:47:02.750 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-240.10.1.el8_3.x86_64 2/5 2021-02-12T17:47:03.097 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-modules-4.18.0-240.10.1.el8_3.x86_64 3/5 2021-02-12T17:47:05.979 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-modules-4.18.0-240.10.1.el8_3.x86_64 3/5 2021-02-12T17:47:06.162 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : kernel-4.18.0-240.10.1.el8_3.x86_64 4/5 2021-02-12T17:47:06.389 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-02-12T17:47:25.911 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: kernel-core-4.18.0-240.10.1.el8_3.x86_64 5/5 2021-02-12T17:47:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-02-12T17:47:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-4.18.0-240.10.1.el8_3.x86_64 1/5 2021-02-12T17:47:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-core-4.18.0-240.10.1.el8_3.x86_64 2/5 2021-02-12T17:47:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : kernel-modules-4.18.0-240.10.1.el8_3.x86_64 3/5 2021-02-12T17:47:26.275 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : linux-firmware-20200619-101.git3890db36.el8_3.noarch 4/5 2021-02-12T17:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : linux-firmware-20191202-97.gite8a0f4c9.el8.noarch 5/5 2021-02-12T17:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2021-02-12T17:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: linux-firmware-20200619-101.git3890db36.el8_3.noarch 2021-02-12T17:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:47:26.653 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-core-4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:47:26.654 INFO:teuthology.orchestra.run.smithi107.stdout: kernel-modules-4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:47:26.654 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:47:26.654 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:47:26.813 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel --last | head -n 1 2021-02-12T17:47:26.931 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-240.10.1.el8_3.x86_64 Fri 12 Feb 2021 05:47:05 PM UTC 2021-02-12T17:47:26.932 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:47:26.932 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-193.28.1.el8_2.x86_64 Expected: 4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:47:26.932 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2021-02-12T17:47:26.932 DEBUG:teuthology.task.kernel:src is distro, skipping download 2021-02-12T17:47:26.933 INFO:teuthology.task.kernel:Installing distro kernel on mon.a... 2021-02-12T17:47:26.933 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi107.front.sepia.ceph.com, path=None, version=distro) 2021-02-12T17:47:26.933 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q kernel --last | head -n 1 2021-02-12T17:47:27.021 INFO:teuthology.orchestra.run.smithi107.stdout:kernel-4.18.0-240.10.1.el8_3.x86_64 Fri 12 Feb 2021 05:47:05 PM UTC 2021-02-12T17:47:27.023 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:47:27.023 DEBUG:teuthology.orchestra.run.smithi107:> sudo rpm -qi grub2-tools 2021-02-12T17:47:27.052 INFO:teuthology.orchestra.run.smithi107.stdout:Name : grub2-tools 2021-02-12T17:47:27.053 INFO:teuthology.orchestra.run.smithi107.stdout:Epoch : 1 2021-02-12T17:47:27.053 INFO:teuthology.orchestra.run.smithi107.stdout:Version : 2.02 2021-02-12T17:47:27.053 INFO:teuthology.orchestra.run.smithi107.stdout:Release : 81.el8 2021-02-12T17:47:27.053 INFO:teuthology.orchestra.run.smithi107.stdout:Architecture: x86_64 2021-02-12T17:47:27.053 INFO:teuthology.orchestra.run.smithi107.stdout:Install Date: Fri 13 Nov 2020 03:02:55 PM UTC 2021-02-12T17:47:27.053 INFO:teuthology.orchestra.run.smithi107.stdout:Group : System Environment/Base 2021-02-12T17:47:27.053 INFO:teuthology.orchestra.run.smithi107.stdout:Size : 9192705 2021-02-12T17:47:27.053 INFO:teuthology.orchestra.run.smithi107.stdout:License : GPLv3+ 2021-02-12T17:47:27.054 INFO:teuthology.orchestra.run.smithi107.stdout:Signature : RSA/SHA256, Tue 14 Apr 2020 03:18:37 PM UTC, Key ID 05b555b38483c65d 2021-02-12T17:47:27.054 INFO:teuthology.orchestra.run.smithi107.stdout:Source RPM : grub2-2.02-81.el8.src.rpm 2021-02-12T17:47:27.054 INFO:teuthology.orchestra.run.smithi107.stdout:Build Date : Tue 14 Apr 2020 02:54:04 PM UTC 2021-02-12T17:47:27.054 INFO:teuthology.orchestra.run.smithi107.stdout:Build Host : kbuilder.bsys.centos.org 2021-02-12T17:47:27.054 INFO:teuthology.orchestra.run.smithi107.stdout:Relocations : (not relocatable) 2021-02-12T17:47:27.054 INFO:teuthology.orchestra.run.smithi107.stdout:Packager : CentOS BuildSystem 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:Vendor : CentOS 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:URL : http://www.gnu.org/software/grub/ 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:Summary : Support tools for GRUB. 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:Description : 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:customizable bootloader with modular architecture. It supports a rich 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:variety of kernel formats, file systems, computer architectures and 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:hardware devices. 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:47:27.055 INFO:teuthology.orchestra.run.smithi107.stdout:This subpackage provides tools for support of all platforms. 2021-02-12T17:47:27.056 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2021-02-12T17:47:27.056 INFO:teuthology.task.kernel:Updating grub on smithi107 to boot 4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:47:27.056 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2021-02-12T17:47:28.772 INFO:teuthology.orchestra.run.smithi107.stderr:Generating grub configuration file ... 2021-02-12T17:47:30.672 INFO:teuthology.orchestra.run.smithi107.stderr:done 2021-02-12T17:47:30.674 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2021-02-12T17:47:30.696 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.k8B61chaVu 2021-02-12T17:47:30.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.k8B61chaVu 2021-02-12T17:47:30.766 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.k8B61chaVu 2021-02-12T17:47:30.914 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.k8B61chaVu is 4KB 2021-02-12T17:47:30.962 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.k8B61chaVu 2021-02-12T17:47:30.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo /bin/ls /boot/loader/entries || true 2021-02-12T17:47:31.044 INFO:teuthology.orchestra.run.smithi107.stdout:2951a06464bf41f6a3bf91594147ecff-0-rescue.conf 2021-02-12T17:47:31.045 INFO:teuthology.orchestra.run.smithi107.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-193.28.1.el8_2.x86_64.conf 2021-02-12T17:47:31.045 INFO:teuthology.orchestra.run.smithi107.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-193.el8.x86_64.conf 2021-02-12T17:47:31.045 INFO:teuthology.orchestra.run.smithi107.stdout:2951a06464bf41f6a3bf91594147ecff-4.18.0-240.10.1.el8_3.x86_64.conf 2021-02-12T17:47:31.046 DEBUG:teuthology.orchestra.run.smithi107:> sudo grub2-set-default 2951a06464bf41f6a3bf91594147ecff-4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:47:31.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo shutdown -r now 2021-02-12T17:48:01.225 INFO:teuthology.misc:Re-opening connections... 2021-02-12T17:48:01.226 INFO:teuthology.misc:trying to connect to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:48:01.226 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:48:01.227 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:48:19.782 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2021-02-12T17:48:19.783 DEBUG:teuthology.misc:waited 18.55695366859436 2021-02-12T17:48:20.783 INFO:teuthology.misc:trying to connect to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:48:20.783 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:48:20.784 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:48:23.842 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2021-02-12T17:48:23.842 DEBUG:teuthology.misc:waited 22.61681365966797 2021-02-12T17:48:24.843 INFO:teuthology.misc:trying to connect to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:48:24.843 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:48:24.844 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:48:26.914 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2021-02-12T17:48:26.914 DEBUG:teuthology.misc:waited 25.688800811767578 2021-02-12T17:48:27.915 INFO:teuthology.misc:trying to connect to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:48:27.915 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:48:27.916 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:48:29.990 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2021-02-12T17:48:29.990 DEBUG:teuthology.misc:waited 28.764781713485718 2021-02-12T17:48:30.991 INFO:teuthology.misc:trying to connect to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:48:30.991 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:48:30.991 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:48:33.062 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2021-02-12T17:48:33.062 DEBUG:teuthology.misc:waited 31.836799383163452 2021-02-12T17:48:34.063 INFO:teuthology.misc:trying to connect to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:48:34.063 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:48:34.063 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:48:36.130 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.107 2021-02-12T17:48:36.130 DEBUG:teuthology.misc:waited 34.90481448173523 2021-02-12T17:48:37.131 INFO:teuthology.misc:trying to connect to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:48:37.131 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:48:37.132 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:49:37.134 DEBUG:teuthology.orchestra.remote:timed out 2021-02-12T17:49:37.134 DEBUG:teuthology.misc:waited 95.9086058139801 2021-02-12T17:49:38.135 INFO:teuthology.misc:trying to connect to ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:49:38.135 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:49:38.135 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:49:38.508 DEBUG:teuthology.orchestra.run.smithi107:> true 2021-02-12T17:49:38.850 DEBUG:teuthology.misc:waited 97.62429141998291 2021-02-12T17:49:39.851 INFO:teuthology.task.kernel:Checking client mon.a for new kernel version... 2021-02-12T17:49:39.852 INFO:teuthology.task.kernel:Checking kernel version of mon.a, want "4.18.0-240.10.1.el8_3.x86_64"... 2021-02-12T17:49:39.852 DEBUG:teuthology.orchestra.run.smithi107:> uname -r 2021-02-12T17:49:39.868 INFO:teuthology.orchestra.run.smithi107.stdout:4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:49:39.868 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-240.10.1.el8_3.x86_64 vs 4.18.0-240.10.1.el8_3.x86_64 2021-02-12T17:49:39.868 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2021-02-12T17:49:39.869 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2021-02-12T17:49:40.870 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2021-02-12T17:49:40.870 DEBUG:teuthology.orchestra.run.smithi107:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2021-02-12T17:49:40.948 INFO:teuthology.orchestra.run.smithi107.stdout:ttyS1 2021-02-12T17:49:41.535 INFO:teuthology.run_tasks:Running task internal.base... 2021-02-12T17:49:41.547 INFO:teuthology.task.internal:Creating test directory... 2021-02-12T17:49:41.548 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2021-02-12T17:49:41.565 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2021-02-12T17:49:41.579 INFO:teuthology.run_tasks:Running task internal.archive... 2021-02-12T17:49:41.628 INFO:teuthology.task.internal:Creating archive directory... 2021-02-12T17:49:41.628 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2021-02-12T17:49:41.651 INFO:teuthology.run_tasks:Running task internal.coredump... 2021-02-12T17:49:41.663 INFO:teuthology.task.internal:Enabling coredump saving... 2021-02-12T17:49:41.663 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2021-02-12T17:49:41.715 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-12T17:49:41.727 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2021-02-12T17:49:41.728 INFO:teuthology.run_tasks:Running task internal.sudo... 2021-02-12T17:49:41.735 INFO:teuthology.task.internal:Configuring sudo... 2021-02-12T17:49:41.736 DEBUG:teuthology.orchestra.run.smithi107:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2021-02-12T17:49:41.795 INFO:teuthology.run_tasks:Running task internal.syslog... 2021-02-12T17:49:41.827 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2021-02-12T17:49:41.827 DEBUG:teuthology.orchestra.run.smithi107:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2021-02-12T17:49:41.851 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-12T17:49:41.929 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2021-02-12T17:49:41.982 DEBUG:teuthology.orchestra.run.smithi107:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-12T17:49:42.054 DEBUG:teuthology.orchestra.run.smithi107:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2021-02-12T17:49:42.102 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:49:42.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2021-02-12T17:49:42.167 DEBUG:teuthology.orchestra.run.smithi107:> sudo service rsyslog restart 2021-02-12T17:49:42.257 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-12T17:49:42.569 INFO:teuthology.run_tasks:Running task internal.timer... 2021-02-12T17:49:42.578 INFO:teuthology.task.internal:Starting timer... 2021-02-12T17:49:42.578 INFO:teuthology.run_tasks:Running task pcp... 2021-02-12T17:49:42.592 INFO:teuthology.run_tasks:Running task selinux... 2021-02-12T17:49:42.611 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2021-02-12T17:49:42.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo service auditd rotate 2021-02-12T17:49:42.690 INFO:teuthology.orchestra.run.smithi107.stdout:Rotating logs: [ OK ] 2021-02-12T17:49:42.692 DEBUG:teuthology.task.selinux:Getting current SELinux state 2021-02-12T17:49:42.692 DEBUG:teuthology.orchestra.run.smithi107:> /usr/sbin/getenforce 2021-02-12T17:49:42.724 INFO:teuthology.orchestra.run.smithi107.stdout:Permissive 2021-02-12T17:49:42.724 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi107.front.sepia.ceph.com': 'permissive'} 2021-02-12T17:49:42.725 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-02-12T17:49:42.789 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-12T17:49:42.789 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2021-02-12T17:49:42.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo /usr/sbin/setenforce permissive 2021-02-12T17:49:42.861 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2021-02-12T17:49:42.870 INFO:teuthology.repo_utils:/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master was just updated or references a specific commit; assuming it is current 2021-02-12T17:49:42.870 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master to origin/master 2021-02-12T17:49:42.885 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}] 2021-02-12T17:49:42.886 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi107.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/cephlab.yml 2021-02-12T17:49:43.225 INFO:teuthology.task.ansible.out:Using /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/ansible.cfg as config file 2021-02-12T17:49:44.123 INFO:teuthology.task.ansible.out:[WARNING]: file /home/teuthworker/src/git.ceph.com_git_ceph-cm- ansible_master/roles/testnode/tasks/setup-opensuse.yml is empty and had no tasks to include 2021-02-12T17:49:46.657 INFO:teuthology.task.ansible.out: PLAY [all] ********************************************************************* 2021-02-12T17:49:46.678 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:46 +0000 (0:00:00.069) 0:00:00.069 ******* 2021-02-12T17:49:48.125 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi107.front.sepia.ceph.com] 2021-02-12T17:49:48.200 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:48 +0000 (0:00:01.522) 0:00:01.591 ******* 2021-02-12T17:49:48.505 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the sudo group.] ******************************** 2021-02-12T17:49:48.507 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-12T17:49:48.542 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:48 +0000 (0:00:00.342) 0:00:01.933 ******* 2021-02-12T17:49:48.938 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the ansible user.] ****************************** 2021-02-12T17:49:48.940 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 1100, "groups": "sudo", "home": "/home/cm", "move_home": false, "name": "cm", "shell": "/bin/bash", "state": "present", "uid": 1100} 2021-02-12T17:49:48.970 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:48 +0000 (0:00:00.428) 0:00:02.361 ******* 2021-02-12T17:49:48.995 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Delete the ansible users password.] ******************** 2021-02-12T17:49:48.996 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:49:49.028 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:49 +0000 (0:00:00.058) 0:00:02.420 ******* 2021-02-12T17:49:49.343 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Ensure includedir is present in sudoers.] ************** 2021-02-12T17:49:49.345 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-12T17:49:49.376 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:49 +0000 (0:00:00.347) 0:00:02.768 ******* 2021-02-12T17:49:50.309 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Create the cephlab_sudo sudoers.d file.] *************** 2021-02-12T17:49:50.310 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "checksum": "6c8e8b7ecc895eb2094cafe75c34b58f183442a7", "dest": "/etc/sudoers.d/cephlab_sudo", "gid": 0, "group": "root", "md5sum": "c3657b9ef874d0ad278742c3210136a0", "mode": "0440", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 182, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613152189.3888705-64752349172299/source", "state": "file", "uid": 0} 2021-02-12T17:49:50.341 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:50 +0000 (0:00:00.964) 0:00:03.732 ******* 2021-02-12T17:49:50.749 INFO:teuthology.task.ansible.out: TASK [ansible-managed : Add authorized keys for the ansible user.] ************* 2021-02-12T17:49:50.749 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "comment": null, "exclusive": true, "follow": false, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmiHwkLkoJZiQs67KfE8jf0CdXjgbbpzKywUleMk4kWC7Wdn+92q/33DXpfS8xjg7opB9GTK3EW2C/RKBpf5RdaVG/5M7tRyUDLLbS3CURUoF6aNQI+8JIuFlsIfhO3Qw4uT7NlJCqsusrSbyLtGd/Im4CzwhvP1Enn6aUfyxv2yV4HoVjXbbIp2hlRXgcW8G/9q0TMuNfX4J6dpj9/PG8BbkKH/NReL2UMtjCGyRdIQvIwJRHMozjcE21824ZatA7nBeecX0MBRS6/A286uj6Z0PgmfJXYw0yw2Z2uSWR1ptIuMEiDBt96Pzn0dvfl7g/7bCi93MBQf/6sH9n4wkN root@typica001\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv 2021-02-12T17:49:50.751 INFO:teuthology.task.ansible.out:4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/cm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cm", "validate_certs": true} 2021-02-12T17:49:50.835 INFO:teuthology.task.ansible.out: PLAY [teuthology] ************************************************************** skipping: no hosts matched 2021-02-12T17:49:50.842 INFO:teuthology.task.ansible.out: PLAY [testnodes] *************************************************************** 2021-02-12T17:49:50.990 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:50 +0000 (0:00:00.649) 0:00:04.382 ******* 2021-02-12T17:49:51.627 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi107.front.sepia.ceph.com] 2021-02-12T17:49:51.701 INFO:teuthology.task.ansible.out:[WARNING]: Using run_once with the free strategy is not currently supported. This task will still be executed for every host in the inventory list. 2021-02-12T17:49:51.702 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:51 +0000 (0:00:00.710) 0:00:05.092 ******* 2021-02-12T17:49:52.129 INFO:teuthology.task.ansible.out: TASK [users : Check keys_repo HEAD sha1] *************************************** 2021-02-12T17:49:52.131 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "git ls-remote https://github.com/ceph/keys HEAD | awk '{ print $1 }'", "delta": "0:00:00.140641", "end": "2021-02-12 17:49:52.110006", "rc": 0, "start": "2021-02-12 17:49:51.969365", "stderr": "", "stderr_lines": [], "stdout": "df8946f08e870e2d2565b79099289ca48bd825f3", "stdout_lines": ["df8946f08e870e2d2565b79099289ca48bd825f3"]} 2021-02-12T17:49:52.162 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:52 +0000 (0:00:00.460) 0:00:05.553 ******* 2021-02-12T17:49:52.354 INFO:teuthology.task.ansible.out: TASK [users : Check host's /keys-repo-sha1 sentinel file] ********************** 2021-02-12T17:49:52.355 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["cat", "/keys-repo-sha1"], "delta": "0:00:00.010574", "end": "2021-02-12 17:49:52.335338", "failed_when_result": false, "rc": 0, "start": "2021-02-12 17:49:52.324764", "stderr": "", "stderr_lines": [], "stdout": "46fd2a03b54c2ce7520f33746fa0586027b558c3", "stdout_lines": ["46fd2a03b54c2ce7520f33746fa0586027b558c3"]} 2021-02-12T17:49:52.414 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:52 +0000 (0:00:00.251) 0:00:05.805 ******* 2021-02-12T17:49:52.442 INFO:teuthology.task.ansible.out: TASK [users : Determine if we can skip users and pubkeys updates] ************** 2021-02-12T17:49:52.443 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:49:52.475 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:52 +0000 (0:00:00.061) 0:00:05.866 ******* 2021-02-12T17:49:52.500 INFO:teuthology.task.ansible.out: TASK [users : Merge extra_admin_users into managed_admin_users] **************** 2021-02-12T17:49:52.502 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:49:52.533 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:52 +0000 (0:00:00.058) 0:00:05.925 ******* 2021-02-12T17:49:52.560 INFO:teuthology.task.ansible.out: TASK [users : Remove managed_admin_users from managed_users] ******************* 2021-02-12T17:49:52.562 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:49:52.592 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:52 +0000 (0:00:00.058) 0:00:05.983 ******* 2021-02-12T17:49:52.618 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_users list] *********************************** 2021-02-12T17:49:52.619 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:49:52.651 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:52 +0000 (0:00:00.058) 0:00:06.042 ******* 2021-02-12T17:49:52.677 INFO:teuthology.task.ansible.out: TASK [users : Filter the managed_admin_users list] ***************************** 2021-02-12T17:49:52.678 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:49:52.709 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:52 +0000 (0:00:00.057) 0:00:06.100 ******* 2021-02-12T17:49:52.899 INFO:teuthology.task.ansible.out: TASK [users : Set UID_MIN to 1001] ********************************************* 2021-02-12T17:49:52.901 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-12T17:49:52.935 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:49:52 +0000 (0:00:00.226) 0:00:06.326 ******* 2021-02-12T17:49:53.168 INFO:teuthology.task.ansible.out: TASK [users : Create all admin users with sudo access.] ************************ 2021-02-12T17:49:53.170 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1101, "groups": "sudo", "home": "/home/andrewschoen", "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "move_home": false, "name": "andrewschoen", "shell": "/bin/bash", "state": "present", "uid": 1101} 2021-02-12T17:49:53.346 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1102, "groups": "sudo", "home": "/home/zack", "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "move_home": false, "name": "zack", "shell": "/bin/bash", "state": "present", "uid": 1102} 2021-02-12T17:49:53.542 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1103, "groups": "sudo", "home": "/home/kdreyer", "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "move_home": false, "name": "kdreyer", "shell": "/bin/bash", "state": "present", "uid": 1103} 2021-02-12T17:49:53.729 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1104, "groups": "sudo", "home": "/home/dmick", "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "move_home": false, "name": "dmick", "shell": "/bin/bash", "state": "present", "uid": 1104} 2021-02-12T17:49:53.915 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1105, "groups": "sudo", "home": "/home/sage", "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "move_home": false, "name": "sage", "shell": "/bin/bash", "state": "present", "uid": 1105} 2021-02-12T17:49:54.102 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1106, "groups": "sudo", "home": "/home/dgalloway", "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "move_home": false, "name": "dgalloway", "shell": "/bin/bash", "state": "present", "uid": 1106} 2021-02-12T17:49:54.288 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1107, "groups": "sudo", "home": "/home/akraitma", "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "move_home": false, "name": "akraitma", "shell": "/bin/bash", "state": "present", "uid": 1107} 2021-02-12T17:49:54.475 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1108, "groups": "sudo", "home": "/home/jcaratzas", "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "move_home": false, "name": "jcaratzas", "shell": "/bin/bash", "state": "present", "uid": 1108} 2021-02-12T17:49:54.671 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1109, "groups": "sudo", "home": "/home/trociny", "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "move_home": false, "name": "trociny", "shell": "/bin/bash", "state": "present", "uid": 1109} 2021-02-12T17:49:54.857 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1110, "groups": "sudo", "home": "/home/smithfarm", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "move_home": false, "name": "smithfarm", "shell": "/bin/bash", "state": "present", "uid": 1110} 2021-02-12T17:49:55.046 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1111, "groups": "sudo", "home": "/home/mbenjamin", "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "move_home": false, "name": "mbenjamin", "shell": "/bin/bash", "state": "present", "uid": 1111} 2021-02-12T17:49:55.233 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1112, "groups": "sudo", "home": "/home/aemerson", "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "move_home": false, "name": "aemerson", "shell": "/bin/bash", "state": "present", "uid": 1112} 2021-02-12T17:49:55.419 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1113, "groups": "sudo", "home": "/home/amaredia", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "move_home": false, "name": "amaredia", "shell": "/bin/bash", "state": "present", "uid": 1113} 2021-02-12T17:49:55.606 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1114, "groups": "sudo", "home": "/home/tserlin", "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "move_home": false, "name": "tserlin", "shell": "/bin/bash", "state": "present", "uid": 1114} 2021-02-12T17:49:55.792 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1115, "groups": "sudo", "home": "/home/dis", "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "move_home": false, "name": "dis", "shell": "/bin/bash", "state": "present", "uid": 1115} 2021-02-12T17:49:55.989 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1116, "groups": "sudo", "home": "/home/gregf", "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "move_home": false, "name": "gregf", "shell": "/bin/bash", "state": "present", "uid": 1116} 2021-02-12T17:49:56.176 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1117, "groups": "sudo", "home": "/home/joshd", "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "move_home": false, "name": "joshd", "shell": "/bin/bash", "state": "present", "uid": 1117} 2021-02-12T17:49:56.364 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1118, "groups": "sudo", "home": "/home/davidz", "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "move_home": false, "name": "davidz", "shell": "/bin/bash", "state": "present", "uid": 1118} 2021-02-12T17:49:56.551 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1119, "groups": "sudo", "home": "/home/cmeno", "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "move_home": false, "name": "cmeno", "shell": "/bin/bash", "state": "present", "uid": 1119} 2021-02-12T17:49:56.738 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1120, "groups": "sudo", "home": "/home/ivancich", "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "move_home": false, "name": "ivancich", "shell": "/bin/bash", "state": "present", "uid": 1120} 2021-02-12T17:49:56.924 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1121, "groups": "sudo", "home": "/home/wusui", "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "move_home": false, "name": "wusui", "shell": "/bin/bash", "state": "present", "uid": 1121} 2021-02-12T17:49:57.111 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1123, "groups": "sudo", "home": "/home/yuriw", "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "move_home": false, "name": "yuriw", "shell": "/bin/bash", "state": "present", "uid": 1123} 2021-02-12T17:49:57.298 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1124, "groups": "sudo", "home": "/home/tamil", "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "move_home": false, "name": "tamil", "shell": "/bin/bash", "state": "present", "uid": 1124} 2021-02-12T17:49:57.484 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1125, "groups": "sudo", "home": "/home/jowilkin", "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "move_home": false, "name": "jowilkin", "shell": "/bin/bash", "state": "present", "uid": 1125} 2021-02-12T17:49:57.670 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1126, "groups": "sudo", "home": "/home/bhubbard", "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "move_home": false, "name": "bhubbard", "shell": "/bin/bash", "state": "present", "uid": 1126} 2021-02-12T17:49:57.856 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1127, "groups": "sudo", "home": "/home/yehudasa", "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "move_home": false, "name": "yehudasa", "shell": "/bin/bash", "state": "present", "uid": 1127} 2021-02-12T17:49:58.052 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1128, "groups": "sudo", "home": "/home/dang", "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "move_home": false, "name": "dang", "shell": "/bin/bash", "state": "present", "uid": 1128} 2021-02-12T17:49:58.239 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1129, "groups": "sudo", "home": "/home/branto", "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "move_home": false, "name": "branto", "shell": "/bin/bash", "state": "present", "uid": 1129} 2021-02-12T17:49:58.425 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1130, "groups": "sudo", "home": "/home/xiaoxichen", "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "move_home": false, "name": "xiaoxichen", "shell": "/bin/bash", "state": "present", "uid": 1130} 2021-02-12T17:49:58.623 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1131, "groups": "sudo", "home": "/home/ffilz", "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "move_home": false, "name": "ffilz", "shell": "/bin/bash", "state": "present", "uid": 1131} 2021-02-12T17:49:58.809 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1132, "groups": "sudo", "home": "/home/joao", "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "move_home": false, "name": "joao", "shell": "/bin/bash", "state": "present", "uid": 1132} 2021-02-12T17:49:58.995 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1133, "groups": "sudo", "home": "/home/nhm", "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "move_home": false, "name": "nhm", "shell": "/bin/bash", "state": "present", "uid": 1133} 2021-02-12T17:49:59.181 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1134, "groups": "sudo", "home": "/home/nwatkins", "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "move_home": false, "name": "nwatkins", "shell": "/bin/bash", "state": "present", "uid": 1134} 2021-02-12T17:49:59.377 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1135, "groups": "sudo", "home": "/home/mkidd", "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "move_home": false, "name": "mkidd", "shell": "/bin/bash", "state": "present", "uid": 1135} 2021-02-12T17:49:59.574 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1136, "groups": "sudo", "home": "/home/jlopez", "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "move_home": false, "name": "jlopez", "shell": "/bin/bash", "state": "present", "uid": 1136} 2021-02-12T17:49:59.760 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1137, "groups": "sudo", "home": "/home/haomaiwang", "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "move_home": false, "name": "haomaiwang", "shell": "/bin/bash", "state": "present", "uid": 1137} 2021-02-12T17:49:59.946 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1138, "groups": "sudo", "home": "/home/jdillaman", "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "move_home": false, "name": "jdillaman", "shell": "/bin/bash", "state": "present", "uid": 1138} 2021-02-12T17:50:00.132 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1139, "groups": "sudo", "home": "/home/kchai", "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "move_home": false, "name": "kchai", "shell": "/bin/bash", "state": "present", "uid": 1139} 2021-02-12T17:50:00.318 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-12T17:50:00.506 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1141, "groups": "sudo", "home": "/home/dfuller", "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "move_home": false, "name": "dfuller", "shell": "/bin/bash", "state": "present", "uid": 1141} 2021-02-12T17:50:00.692 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1142, "groups": "sudo", "home": "/home/owasserm", "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "move_home": false, "name": "owasserm", "shell": "/bin/bash", "state": "present", "uid": 1142} 2021-02-12T17:50:00.889 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1143, "groups": "sudo", "home": "/home/vasu", "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "move_home": false, "name": "vasu", "shell": "/bin/bash", "state": "present", "uid": 1143} 2021-02-12T17:50:01.075 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1144, "groups": "sudo", "home": "/home/smohan", "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "move_home": false, "name": "smohan", "shell": "/bin/bash", "state": "present", "uid": 1144} 2021-02-12T17:50:01.262 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1145, "groups": "sudo", "home": "/home/abhi", "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "move_home": false, "name": "abhi", "shell": "/bin/bash", "state": "present", "uid": 1145} 2021-02-12T17:50:01.448 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1146, "groups": "sudo", "home": "/home/cbodley", "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "move_home": false, "name": "cbodley", "shell": "/bin/bash", "state": "present", "uid": 1146} 2021-02-12T17:50:01.755 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1147, "groups": "sudo", "home": "/home/mwatts", "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "move_home": false, "name": "mwatts", "shell": "/bin/bash", "state": "present", "uid": 1147} 2021-02-12T17:50:01.830 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-12T17:50:02.017 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1149, "groups": "sudo", "home": "/home/pdonnell", "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "move_home": false, "name": "pdonnell", "shell": "/bin/bash", "state": "present", "uid": 1149} 2021-02-12T17:50:02.204 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1150, "groups": "sudo", "home": "/home/jlayton", "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "move_home": false, "name": "jlayton", "shell": "/bin/bash", "state": "present", "uid": 1150} 2021-02-12T17:50:02.390 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1151, "groups": "sudo", "home": "/home/rzarzynski", "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "move_home": false, "name": "rzarzynski", "shell": "/bin/bash", "state": "present", "uid": 1151} 2021-02-12T17:50:02.577 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1152, "groups": "sudo", "home": "/home/rdias", "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "move_home": false, "name": "rdias", "shell": "/bin/bash", "state": "present", "uid": 1152} 2021-02-12T17:50:02.774 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1153, "groups": "sudo", "home": "/home/vshankar", "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "move_home": false, "name": "vshankar", "shell": "/bin/bash", "state": "present", "uid": 1153} 2021-02-12T17:50:02.960 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1154, "groups": "sudo", "home": "/home/akupczyk", "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "move_home": false, "name": "akupczyk", "shell": "/bin/bash", "state": "present", "uid": 1154} 2021-02-12T17:50:03.148 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1155, "groups": "sudo", "home": "/home/nojha", "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "move_home": false, "name": "nojha", "shell": "/bin/bash", "state": "present", "uid": 1155} 2021-02-12T17:50:03.334 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1156, "groups": "sudo", "home": "/home/ifed01", "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "move_home": false, "name": "ifed01", "shell": "/bin/bash", "state": "present", "uid": 1156} 2021-02-12T17:50:03.522 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1157, "groups": "sudo", "home": "/home/myoungwon", "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "move_home": false, "name": "myoungwon", "shell": "/bin/bash", "state": "present", "uid": 1157} 2021-02-12T17:50:03.719 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1158, "groups": "sudo", "home": "/home/gabrioux", "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "move_home": false, "name": "gabrioux", "shell": "/bin/bash", "state": "present", "uid": 1158} 2021-02-12T17:50:03.905 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1159, "groups": "sudo", "home": "/home/leseb", "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "move_home": false, "name": "leseb", "shell": "/bin/bash", "state": "present", "uid": 1159} 2021-02-12T17:50:04.101 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1160, "groups": "sudo", "home": "/home/hchen", "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "move_home": false, "name": "hchen", "shell": "/bin/bash", "state": "present", "uid": 1160} 2021-02-12T17:50:04.289 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1161, "groups": "sudo", "home": "/home/jcollin", "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "move_home": false, "name": "jcollin", "shell": "/bin/bash", "state": "present", "uid": 1161} 2021-02-12T17:50:04.477 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1162, "groups": "sudo", "home": "/home/xxg", "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "move_home": false, "name": "xxg", "shell": "/bin/bash", "state": "present", "uid": 1162} 2021-02-12T17:50:04.663 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1163, "groups": "sudo", "home": "/home/pcuzner", "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "move_home": false, "name": "pcuzner", "shell": "/bin/bash", "state": "present", "uid": 1163} 2021-02-12T17:50:04.859 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1164, "groups": "sudo", "home": "/home/mkogan", "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "move_home": false, "name": "mkogan", "shell": "/bin/bash", "state": "present", "uid": 1164} 2021-02-12T17:50:05.045 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1140, "groups": "sudo", "home": "/home/vumrao", "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "move_home": false, "name": "vumrao", "shell": "/bin/bash", "state": "present", "uid": 1140} 2021-02-12T17:50:05.232 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1165, "groups": "sudo", "home": "/home/henrix", "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "move_home": false, "name": "henrix", "shell": "/bin/bash", "state": "present", "uid": 1165} 2021-02-12T17:50:05.418 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1166, "groups": "sudo", "home": "/home/pbs1108", "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "move_home": false, "name": "pbs1108", "shell": "/bin/bash", "state": "present", "uid": 1166} 2021-02-12T17:50:05.605 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1167, "groups": "sudo", "home": "/home/clacroix", "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "move_home": false, "name": "clacroix", "shell": "/bin/bash", "state": "present", "uid": 1167} 2021-02-12T17:50:05.791 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1168, "groups": "sudo", "home": "/home/epuertat", "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "move_home": false, "name": "epuertat", "shell": "/bin/bash", "state": "present", "uid": 1168} 2021-02-12T17:50:05.988 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1169, "groups": "sudo", "home": "/home/tdehler", "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "move_home": false, "name": "tdehler", "shell": "/bin/bash", "state": "present", "uid": 1169} 2021-02-12T17:50:06.175 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1170, "groups": "sudo", "home": "/home/laura", "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "move_home": false, "name": "laura", "shell": "/bin/bash", "state": "present", "uid": 1170} 2021-02-12T17:50:06.362 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1171, "groups": "sudo", "home": "/home/yaarit", "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "move_home": false, "name": "yaarit", "shell": "/bin/bash", "state": "present", "uid": 1171} 2021-02-12T17:50:06.549 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1172, "groups": "sudo", "home": "/home/rpavani1998", "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "move_home": false, "name": "rpavani1998", "shell": "/bin/bash", "state": "present", "uid": 1172} 2021-02-12T17:50:06.735 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1173, "groups": "sudo", "home": "/home/rishabh", "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "move_home": false, "name": "rishabh", "shell": "/bin/bash", "state": "present", "uid": 1173} 2021-02-12T17:50:06.922 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1174, "groups": "sudo", "home": "/home/skrah", "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "move_home": false, "name": "skrah", "shell": "/bin/bash", "state": "present", "uid": 1174} 2021-02-12T17:50:07.109 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1175, "groups": "sudo", "home": "/home/smanjara", "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "move_home": false, "name": "smanjara", "shell": "/bin/bash", "state": "present", "uid": 1175} 2021-02-12T17:50:07.296 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1176, "groups": "sudo", "home": "/home/bengland", "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "move_home": false, "name": "bengland", "shell": "/bin/bash", "state": "present", "uid": 1176} 2021-02-12T17:50:07.483 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1177, "groups": "sudo", "home": "/home/pnawracay", "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "move_home": false, "name": "pnawracay", "shell": "/bin/bash", "state": "present", "uid": 1177} 2021-02-12T17:50:07.669 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1178, "groups": "sudo", "home": "/home/alfonsomthd", "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "move_home": false, "name": "alfonsomthd", "shell": "/bin/bash", "state": "present", "uid": 1178} 2021-02-12T17:50:07.865 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1179, "groups": "sudo", "home": "/home/oliveiradan", "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "move_home": false, "name": "oliveiradan", "shell": "/bin/bash", "state": "present", "uid": 1179} 2021-02-12T17:50:08.051 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1180, "groups": "sudo", "home": "/home/swagner", "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "move_home": false, "name": "swagner", "shell": "/bin/bash", "state": "present", "uid": 1180} 2021-02-12T17:50:08.237 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1181, "groups": "sudo", "home": "/home/yuvalif", "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "move_home": false, "name": "yuvalif", "shell": "/bin/bash", "state": "present", "uid": 1181} 2021-02-12T17:50:08.423 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1182, "groups": "sudo", "home": "/home/kkeithle", "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "move_home": false, "name": "kkeithle", "shell": "/bin/bash", "state": "present", "uid": 1182} 2021-02-12T17:50:08.611 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1183, "groups": "sudo", "home": "/home/emmericp", "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "move_home": false, "name": "emmericp", "shell": "/bin/bash", "state": "present", "uid": 1183} 2021-02-12T17:50:08.797 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1184, "groups": "sudo", "home": "/home/mchangir", "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "move_home": false, "name": "mchangir", "shell": "/bin/bash", "state": "present", "uid": 1184} 2021-02-12T17:50:08.995 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1185, "groups": "sudo", "home": "/home/sidharthanup", "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "move_home": false, "name": "sidharthanup", "shell": "/bin/bash", "state": "present", "uid": 1185} 2021-02-12T17:50:09.181 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1186, "groups": "sudo", "home": "/home/varsha", "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "move_home": false, "name": "varsha", "shell": "/bin/bash", "state": "present", "uid": 1186} 2021-02-12T17:50:09.378 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1187, "groups": "sudo", "home": "/home/sjust", "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "move_home": false, "name": "sjust", "shell": "/bin/bash", "state": "present", "uid": 1187} 2021-02-12T17:50:09.565 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1188, "groups": "sudo", "home": "/home/ideepika", "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "move_home": false, "name": "ideepika", "shell": "/bin/bash", "state": "present", "uid": 1188} 2021-02-12T17:50:09.751 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1189, "groups": "sudo", "home": "/home/gsalomon", "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "move_home": false, "name": "gsalomon", "shell": "/bin/bash", "state": "present", "uid": 1189} 2021-02-12T17:50:09.938 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1190, "groups": "sudo", "home": "/home/soumyakoduri", "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "move_home": false, "name": "soumyakoduri", "shell": "/bin/bash", "state": "present", "uid": 1190} 2021-02-12T17:50:10.124 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1191, "groups": "sudo", "home": "/home/kyr", "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "move_home": false, "name": "kyr", "shell": "/bin/bash", "state": "present", "uid": 1191} 2021-02-12T17:50:10.311 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1192, "groups": "sudo", "home": "/home/sseshasa", "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "move_home": false, "name": "sseshasa", "shell": "/bin/bash", "state": "present", "uid": 1192} 2021-02-12T17:50:10.508 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1193, "groups": "sudo", "home": "/home/rfriedma", "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "move_home": false, "name": "rfriedma", "shell": "/bin/bash", "state": "present", "uid": 1193} 2021-02-12T17:50:10.694 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1194, "groups": "sudo", "home": "/home/lmb", "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "move_home": false, "name": "lmb", "shell": "/bin/bash", "state": "present", "uid": 1194} 2021-02-12T17:50:10.880 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1195, "groups": "sudo", "home": "/home/ksirivad", "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "move_home": false, "name": "ksirivad", "shell": "/bin/bash", "state": "present", "uid": 1195} 2021-02-12T17:50:11.067 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1196, "groups": "sudo", "home": "/home/ofriedma", "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "move_home": false, "name": "ofriedma", "shell": "/bin/bash", "state": "present", "uid": 1196} 2021-02-12T17:50:11.253 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1197, "groups": "sudo", "home": "/home/h4iresis", "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "move_home": false, "name": "h4iresis", "shell": "/bin/bash", "state": "present", "uid": 1197} 2021-02-12T17:50:11.440 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1198, "groups": "sudo", "home": "/home/shyukri", "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "move_home": false, "name": "shyukri", "shell": "/bin/bash", "state": "present", "uid": 1198} 2021-02-12T17:50:11.637 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1199, "groups": "sudo", "home": "/home/scarvalhojr", "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "move_home": false, "name": "scarvalhojr", "shell": "/bin/bash", "state": "present", "uid": 1199} 2021-02-12T17:50:11.823 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1200, "groups": "sudo", "home": "/home/rakesh", "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "move_home": false, "name": "rakesh", "shell": "/bin/bash", "state": "present", "uid": 1200} 2021-02-12T17:50:12.010 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1201, "groups": "sudo", "home": "/home/ryg-", "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "move_home": false, "name": "ryg-", "shell": "/bin/bash", "state": "present", "uid": 1201} 2021-02-12T17:50:12.206 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1148, "groups": "sudo", "home": "/home/prsrivas", "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "move_home": false, "name": "prsrivas", "shell": "/bin/bash", "state": "present", "uid": 1148} 2021-02-12T17:50:12.392 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1202, "groups": "sudo", "home": "/home/zdover", "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "move_home": false, "name": "zdover", "shell": "/bin/bash", "state": "present", "uid": 1202} 2021-02-12T17:50:12.579 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1203, "groups": "sudo", "home": "/home/rraja", "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "move_home": false, "name": "rraja", "shell": "/bin/bash", "state": "present", "uid": 1203} 2021-02-12T17:50:12.765 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1204, "groups": "sudo", "home": "/home/toabctl", "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "move_home": false, "name": "toabctl", "shell": "/bin/bash", "state": "present", "uid": 1204} 2021-02-12T17:50:12.951 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1205, "groups": "sudo", "home": "/home/mgfritch", "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "move_home": false, "name": "mgfritch", "shell": "/bin/bash", "state": "present", "uid": 1205} 2021-02-12T17:50:13.148 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1206, "groups": "sudo", "home": "/home/xiubli", "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "move_home": false, "name": "xiubli", "shell": "/bin/bash", "state": "present", "uid": 1206} 2021-02-12T17:50:13.334 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1207, "groups": "sudo", "home": "/home/jafaj", "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "move_home": false, "name": "jafaj", "shell": "/bin/bash", "state": "present", "uid": 1207} 2021-02-12T17:50:13.521 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1208, "groups": "sudo", "home": "/home/jschmid", "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "move_home": false, "name": "jschmid", "shell": "/bin/bash", "state": "present", "uid": 1208} 2021-02-12T17:50:13.708 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1209, "groups": "sudo", "home": "/home/kachau", "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "move_home": false, "name": "kachau", "shell": "/bin/bash", "state": "present", "uid": 1209} 2021-02-12T17:50:13.894 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1210, "groups": "sudo", "home": "/home/khiremat", "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "move_home": false, "name": "khiremat", "shell": "/bin/bash", "state": "present", "uid": 1210} 2021-02-12T17:50:14.092 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1211, "groups": "sudo", "home": "/home/kapandya", "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "move_home": false, "name": "kapandya", "shell": "/bin/bash", "state": "present", "uid": 1211} 2021-02-12T17:50:14.279 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1212, "groups": "sudo", "home": "/home/prasad", "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "move_home": false, "name": "prasad", "shell": "/bin/bash", "state": "present", "uid": 1212} 2021-02-12T17:50:14.466 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1213, "groups": "sudo", "home": "/home/vikhyat", "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "move_home": false, "name": "vikhyat", "shell": "/bin/bash", "state": "present", "uid": 1213} 2021-02-12T17:50:14.664 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1214, "groups": "sudo", "home": "/home/shraddhaag", "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "move_home": false, "name": "shraddhaag", "shell": "/bin/bash", "state": "present", "uid": 1214} 2021-02-12T17:50:14.851 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1215, "groups": "sudo", "home": "/home/gkyratsas", "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "move_home": false, "name": "gkyratsas", "shell": "/bin/bash", "state": "present", "uid": 1215} 2021-02-12T17:50:15.038 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1216, "groups": "sudo", "home": "/home/prateek", "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "move_home": false, "name": "prateek", "shell": "/bin/bash", "state": "present", "uid": 1216} 2021-02-12T17:50:15.225 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1217, "groups": "sudo", "home": "/home/jelopez", "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "move_home": false, "name": "jelopez", "shell": "/bin/bash", "state": "present", "uid": 1217} 2021-02-12T17:50:15.422 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1218, "groups": "sudo", "home": "/home/dcain", "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "move_home": false, "name": "dcain", "shell": "/bin/bash", "state": "present", "uid": 1218} 2021-02-12T17:50:15.608 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1219, "groups": "sudo", "home": "/home/sunny", "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "move_home": false, "name": "sunny", "shell": "/bin/bash", "state": "present", "uid": 1219} 2021-02-12T17:50:15.795 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1220, "groups": "sudo", "home": "/home/xuxuehan", "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "move_home": false, "name": "xuxuehan", "shell": "/bin/bash", "state": "present", "uid": 1220} 2021-02-12T17:50:15.992 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1221, "groups": "sudo", "home": "/home/aclewett", "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "move_home": false, "name": "aclewett", "shell": "/bin/bash", "state": "present", "uid": 1221} 2021-02-12T17:50:16.179 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1222, "groups": "sudo", "home": "/home/cblum", "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "move_home": false, "name": "cblum", "shell": "/bin/bash", "state": "present", "uid": 1222} 2021-02-12T17:50:16.376 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1223, "groups": "sudo", "home": "/home/mlatimer", "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "move_home": false, "name": "mlatimer", "shell": "/bin/bash", "state": "present", "uid": 1223} 2021-02-12T17:50:16.562 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1224, "groups": "sudo", "home": "/home/shyamsundarr", "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "move_home": false, "name": "shyamsundarr", "shell": "/bin/bash", "state": "present", "uid": 1224} 2021-02-12T17:50:16.749 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1225, "groups": "sudo", "home": "/home/shuzhenyi", "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "move_home": false, "name": "shuzhenyi", "shell": "/bin/bash", "state": "present", "uid": 1225} 2021-02-12T17:50:16.946 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1226, "groups": "sudo", "home": "/home/mdipalma", "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "move_home": false, "name": "mdipalma", "shell": "/bin/bash", "state": "present", "uid": 1226} 2021-02-12T17:50:17.133 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1227, "groups": "sudo", "home": "/home/kchang", "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "move_home": false, "name": "kchang", "shell": "/bin/bash", "state": "present", "uid": 1227} 2021-02-12T17:50:17.320 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1228, "groups": "sudo", "home": "/home/vereddy", "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "move_home": false, "name": "vereddy", "shell": "/bin/bash", "state": "present", "uid": 1228} 2021-02-12T17:50:17.517 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1229, "groups": "sudo", "home": "/home/benhanokh", "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "move_home": false, "name": "benhanokh", "shell": "/bin/bash", "state": "present", "uid": 1229} 2021-02-12T17:50:17.704 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1230, "groups": "sudo", "home": "/home/jsalomon", "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "move_home": false, "name": "jsalomon", "shell": "/bin/bash", "state": "present", "uid": 1230} 2021-02-12T17:50:17.890 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1231, "groups": "sudo", "home": "/home/jolmomar", "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "move_home": false, "name": "jolmomar", "shell": "/bin/bash", "state": "present", "uid": 1231} 2021-02-12T17:50:18.076 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1232, "groups": "sudo", "home": "/home/hmunjulu", "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "move_home": false, "name": "hmunjulu", "shell": "/bin/bash", "state": "present", "uid": 1232} 2021-02-12T17:50:18.273 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1233, "groups": "sudo", "home": "/home/julpark", "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "move_home": false, "name": "julpark", "shell": "/bin/bash", "state": "present", "uid": 1233} 2021-02-12T17:50:18.459 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "append": true, "changed": false, "comment": "", "group": 1235, "groups": "sudo", "home": "/home/pdhange", "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "move_home": false, "name": "pdhange", "shell": "/bin/bash", "state": "present", "uid": 1235} 2021-02-12T17:50:20.201 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10102, "groups": "sudo", "home": "/home/dpivonka", "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "name": "dpivonka", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10102} 2021-02-12T17:50:21.480 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10103, "groups": "sudo", "home": "/home/adking", "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "name": "adking", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10103} 2021-02-12T17:50:22.748 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10104, "groups": "sudo", "home": "/home/amathuri", "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "name": "amathuri", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10104} 2021-02-12T17:50:24.210 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": "", "create_home": true, "group": 10105, "groups": "sudo", "home": "/home/nizamial09", "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "name": "nizamial09", "shell": "/bin/bash", "state": "present", "system": false, "uid": 10105} 2021-02-12T17:50:24.323 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:50:24 +0000 (0:00:31.388) 0:00:37.714 ******* 2021-02-12T17:50:24.349 INFO:teuthology.task.ansible.out: TASK [users : Create all users without sudo access.] *************************** 2021-02-12T17:50:24.382 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:50:24 +0000 (0:00:00.058) 0:00:37.773 ******* 2021-02-12T17:50:24.429 INFO:teuthology.task.ansible.out: TASK [users : Merge managed_users and managed_admin users] ********************* 2021-02-12T17:50:24.431 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"pubkey_users": [{"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, {"name": "jcaratzas", "ovpn": "jcaratza@fedora 2021-02-12T17:50:24.433 INFO:teuthology.task.ansible.out:rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265 2021-02-12T17:50:24.434 INFO:teuthology.task.ansible.out:c63f3ea91e7dacd78"}, {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, {"name": "ivancic 2021-02-12T17:50:24.435 INFO:teuthology.task.ansible.out:h", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1Wi 2021-02-12T17:50:24.437 INFO:teuthology.task.ansible.out:RenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, {"name": "jdillaman", "ovpn": "jdilla 2021-02-12T17:50:24.437 INFO:teuthology.task.ansible.out:man@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, {"name": "vumrao", "ovpn": "vumrao@vum 2021-02-12T17:50:24.438 INFO:teuthology.task.ansible.out:rao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69 2021-02-12T17:50:24.439 INFO:teuthology.task.ansible.out:mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d8 2021-02-12T17:50:24.441 INFO:teuthology.task.ansible.out:41a16dda2504cca6d95df831a652a30d2e5655ed"}, {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, {"name": "oliveiradan", "ovpn": "oliveirada 2021-02-12T17:50:24.442 INFO:teuthology.task.ansible.out:n@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOe 2021-02-12T17:50:24.443 INFO:teuthology.task.ansible.out:nVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, {"name": "rraja", "o 2021-02-12T17:50:24.443 INFO:teuthology.task.ansible.out:vpn": "rraja@p50 4NMUghtas9 2021-02-12T17:50:24.445 INFO:teuthology.task.ansible.out:gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, {"name": 2021-02-12T17:50:24.446 INFO:teuthology.task.ansible.out:"dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NI 2021-02-12T17:50:24.448 INFO:teuthology.task.ansible.out:Nj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}]}, "changed": false} 2021-02-12T17:50:24.483 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:50:24 +0000 (0:00:00.100) 0:00:37.874 ******* 2021-02-12T17:50:25.241 INFO:teuthology.task.ansible.out: TASK [users : Clone the keys repo] ********************************************* 2021-02-12T17:50:25.242 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com -> localhost] => {"after": "df8946f08e870e2d2565b79099289ca48bd825f3", "attempts": 1, "before": "df8946f08e870e2d2565b79099289ca48bd825f3", "changed": false, "remote_url_changed": false} 2021-02-12T17:50:25.275 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:50:25 +0000 (0:00:00.792) 0:00:38.666 ******* 2021-02-12T17:50:25.601 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys using the keys repo] ********************** 2021-02-12T17:50:25.603 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDehvF7TbkfX9Ga4fNdq/j9GHu3c9cNk4iIHVs7e14j0h9DlV14aPyn/M0+z90PK+5p2uL9EvRzkbsPrfxpG3pxUa1pmVmKIPvBruA7ln8tExOVwO3XEsBG7b8VWUMlQmwRS9WEfkKAIHlr9cgIFGb1yrMW30RyL7keO1giotAq/wR9pMicyqYAbZOtI3MuPsDVTNxtyD3RwRUQicz6vNaK/RwfMCZVgdvoinrJ7VSyFVEiitzuQqMlrF935sfLvhuyxmwvJKfZI8QMEYcFbKd1cDnCk9n/PySNRY96gb41nb5pUk+0eaDu37jnmj7LLqoidD3UWUuHNnGBAAk+zNlD andrew.schoen@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDdkebJGXKQWBG/7wZG334P+EAdC+5VNMrjLjXhaZZbt3V9vKDQCodcU3zSOjMa/GDGg2WHQdO/vioCc6O/m61KTbgLDaaPSzHH16j/e1SaU8BIgKKU42AUBJzpPeaZ2wCC1i12Hq8ULwjEW29IDsnMortzlCSQmaTKc3DN2nCXZWQXA1rSiQ2AhIq5WNJY9S36lsg5tkex4xUajyNW4vPubLtkSv+WrzNK7ZJ8zfzP6eJEJ+Uj7CLdanBCgwoLGSPfGA5c/gyUT7mHibOUwvFB5sFVsozMPp2PXzUUIv70adJpOskLYfKY7TUSuaB4AUOPu6NVauzVIl3Tsn+uvHqvtn8KC3BPpyH7+JiFQWEt+aO+/ynprfoqCjktuf07SXo0ewoVYT2S++JEBa/N2hKRkzIoSPJ+Rvoazc8GZIjXpQRyhMdTV/LCSeyUTJ7lDQVrQVhUIHKnMm+hZmf4tLQKZvNAThO984CpY93ST5Ww07fjRLMXNZGuaz+zHKNexhWSPKr+ggChnZCOLl/F6zbDnBZoCaHR6hH4+QLz0K/FAgxN2Nn00RiOFkmIlzEioxMcwC/peTl3WzRfqPINdTftiTzcFi/c2QmlHhnxsueIwNhHOn4frkGkFmexdDrXRStERzSBJ13X4kWduPmQsn+bx5/7xwqC6M4bXpkYFOefOw== aschoen@redhat.com", "key_options": null, "keyfile": "/home/andrewschoen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "andrewschoen", "validate_certs": true} 2021-02-12T17:50:25.809 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBwJCoMbbxQrI9jM5I/lX0MYDB57LzYXDJTXqmLfJ8E5IHwQbn708/EFXAHuJlCyoJAuA8FaU/Y5/l2LBlVXwp9v+8ftk/D6AcsuQ+Hi0ZLCTvW10OUIS2cwX/of//JJIu6roy3r3SWwUcQvwjaZYLm7QkC+1StsLNMHdgiVRFiibqml72Q53VTtSTt6z2aXjrHzhpZq0hBK/13aknzAjHvGprQnchKVzhA/7A5pArF3CXpnI7aCwKvvjU0AcvqqD8WNLMpF4hFSoTK7xluKB+5UAJ36wefIO2KFM/zMwUtl4/aDhDdPeSaQRWq/q7viKLVNS0X9j2PjSRTfYhJNZr zack@zack-cerzas-macbook-pro.local", "key_options": null, "keyfile": "/home/zack/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zack", "validate_certs": true} 2021-02-12T17:50:26.016 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4eHmz10szeHNS3dNejKokW85ksB+iR4HGOFsmQM11Ni68Nm5aqEKvkOZU8TpY92vpCQL0A68GlrXB845cACdyk6HUJYyNNNMC43l1FYWOwjMqQBSdj8W3VQDTA6eiG60mt5fgI8fyR38rKzIA1MnTBkSSjuh5kQVJ9bdEp3GuY5oc8vxDNBlGJ6LYnyEWt/pqL2J+mpjqnOjsC+EbE2exhP9O+mvzpQiyo/+dEN1COwX3//pNRXGfOSeOczHNsJE8Eu+j/n/BlW57++sJyFMkzS7bUxMSGM6quvjQZ7RT1c5JM6vLEiQyzQxoRgzY93h1yKlOstBi0NamtpqHQZGP kdreyer@mbp.ktdreyer.com", "key_options": null, "keyfile": "/home/kdreyer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kdreyer", "validate_certs": true} 2021-02-12T17:50:26.223 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC305rmql/Qm8Si+DFvAs77hSUZakwiXhhAhSetS89bWq3AdPQTL1MUXJHFm3AMzwzb00tndvsl1AUmFseXLfAi87daNcKJdwEj/JvXgcHKzAXOpVRfArHh08JnOumQiP4EoLYvq7rL+3LjEFURKOWJLLnGxh4E+iiJAnxLTG3WuUcnz9kXVTwtmIs60+2yH+gZJkkB04mnivPYjX46sHigoKgtlE2JnQfOU1lBkeZaBeUhoZX2rIKwRyGELPPcUBwHk10izyk6tePLAw9bSWeS9uEaa/m+qa8neJDyR4/hWQNR9yRXp0YjKkiKoFog+qtsBt3xBNzTPsFLxZ5zugmp dmick@angus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/onMoOCUhkp+UiJAq+ye4P70sfeV+uMhgWKLfLBh/1SYq1xxb9bSmjBnETO/9KvukRhzlReBQ7y+BKRhm037P7LuTUY4r0Y8JB6BXkjjCY1yXZJC6kvB7xFsvyiDum5rrdeWMQ/Rhh7YJpze5KY1nHLkyU9X7Ewak0kLEyduJEKekbsOom6l0wZ2ZYrBEKFWJXUDk5icUuWWg3mhKaKnLNWVg1lUw8jzYTJYfjR9GUgzZWXtzkAaI3ugpLOlQ+KEls03/WAFcWA9F9ExqzIsJpvEks27LeFBGVE3PjVNKMqqBZQlZ 2021-02-12T17:50:26.225 INFO:teuthology.task.ansible.out:tIEz1iguWJaWMi5lDIfXkvVOFabtIpX/BZRf dmick@dmick-MacBookAir\nssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIOoIm6sOytKwqNf9FSQn0Olu72ZRDjAYeNFQVWTn2PIB ed25519-hplaptop", "key_options": null, "keyfile": "/home/dmick/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dmick", "validate_certs": true} 2021-02-12T17:50:26.430 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJ4z0gLwjexgyixdTuiwaxpPnq7w4FOKxHsQKKAHTlKRu11ynB+9APbtqaAYss8yDqHaHyewPdE+9JMYhZX8rUgcKv2H7jBCQ/Enfl0UyY643VQnuG8lr2Ipk6ELuL5NuuTb7Npo4zKyq/0649kPD46k/X/qLuKzhN49k50S3XJq5WwHi6CJvR8DzPOkjied0o1BHpfcCCIMorUcyfVxqmeu2m+uSE15ncQsd8BIvH4/YNoncUGYxqVhGB+FNUlWdhV0SPRW7V/gmyvFW5qdYX2QhNZAQHChdEH1E9AI7WiCFkyLRx39qYgXgqj8vOOnlwHgy/oJnpnu/Mlvj1uETX sage@montreal\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCVPfoCpv2/9uWCwEo4clWt3i1O8ZPpB2FcQbBLxY3iLQWZJDe74rNWPBj0cJVCytTHxYV0e5HxYyChOG60ca5odHlPNB+/Z4WLU9LVuHiCYsElivJlwvD+l9kzI+UMGJb3HCHqx2yIRGXpEkSMFR1+w5Oz9sInPD7JJmO8Hb9KkoeqZtkJL0aic/M5bDWBut6SSfYlmh8qnQYbkCDiXg/KtOcy1ZJXQ9ZLjsCbpSzeBNbdUOxEQ1+5vUe00vHpl0vOKtRrzgTne2ct1IxbSjdL16ZgHi5BQAREXc9UUG3z7NPOmnwkuKkkpyLmwcOusot5tYHrxks+5i6EoGEJrvJ3 sage@maetl", "key_options": null, "keyfile": "/home/sage/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sage", "validate_certs": true} 2021-02-12T17:50:26.648 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBAN3MlkYQNsm1uB6uJQ7hWJ1Ro9Gusrhi/b/KshPMFh51VWBQn71LdCCjdrsE6FWs5FYE/hr7MITJe+si7aZCmuf8U9tc+oiYh3spXtRhJVIM80FOVRXUmt2XR8I+oNbpcFDayYu956Volafksk3oYjsO356swj7+Q+AUQVF4bDJRAAAAFQCkB2YZKItSDL7XfAl6ZmJ21vqUMwAAAIEAsZspNIUwssVJGO4r3M3BECWklWLEFviIjGVKdPWulN10eRCczZrH7vMOs8coHKDQqYdS2RejlojT03FjFAqeBiTeAlcmgHoev+vqTg2Yhqx3gROzNc6NmG887owCOU4j7LJQP6SX8I8EpSXCMNOZliv2ZokjX2DuRsuIDxQdc84AAACBAMAp++fcqmGS7x7ic2x4rhyiYmdqlMIl8lrM3GkVR6qGuIVBvOGl38ieJKcTBLeRgyC04uIMfk8P9lLvZw3UF97qH3V0imKfq8aYnBGxCv+IjtdOrR5k4hrUW+P0qipvU+4dgHlpqwTKB4MtD9uDxvhENWvqcCnOS5rI6dddaZT6 dgallowa@dgallowa.csb\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKT1KilDopHJzDZ90QJx4+7WVIC4WNdvpD6VrS83B3PFOplNyXZLfDSDRZ4wEzIKLwtR9vLWWInVBNFU00W/2dObSC+l9GDzKsHtO3KSVmAjHEk6ZAJ5Mc3eUPR0lAywpFD8+yg+kDDIkRtCtoHDq+kFvNQutdvrMuhNWOZLyQ2N2XrpRJVD2kiJvaml6nQoZXMtVQGPVcYbn3TRed/wxT/han2u6j/wiElv4r9ty+9zi3P1MoNiALD53sD7wMr/5Si6PX+5zWcXTNcQmTuGpiLwGw0RxCwr4gtwqKBaNae2d1bwnHanLc1KvQEw29O6aBca2f7GWxFY6QtQV/mNft dgalloway@w541\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoEI+g74GcJe/03nDypMzDWqg93K1h13xQPFvCIN7ElNZ+ulY5av5KLnv/gmLbYGRPlgxC7LNZXE7YQPUo2DfA8awnDkAkE/nWvL4fQjUxmaiPthjIWsKvk9EvPIqzLjDbwsiLiRPE0Sxka0qBa1GHU1QYN6cw7l9nWF0dG/TsKw+QmuArBKMgm9Th/QVLq1ARHC/07M1n195QvBjdrKSzOUl+2L1tADHg++sEWDq/MJMiylyNdxp79Wz7aTIlR6l8NV+Ega9LUb6B6xmO7k+J+/hJF7gYOeFl+YqnZttEdz+ 2021-02-12T17:50:26.649 INFO:teuthology.task.ansible.out:o8FbLjhTjOF9QkBsGWlrawXtQln/cVCryV+LAt57L dgalloway@p50", "key_options": null, "keyfile": "/home/dgalloway/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dgalloway", "validate_certs": true} 2021-02-12T17:50:26.858 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo3gswIiZ2xPWMUQNjV1Gd4gYSxVpJM8Jb5pH1lqvoKAsmhVI30qTYxIaezzDWfSYwdkNYQAC9eM5v7EMQrOI7eZtTzCH+gbF79GciGCLL2bDze1I8hJ+jUu4bv+wnmvKsHyyOVBBJPSJ+eGFCzsu7vOiK4iFX0zgCXYwvP+A4Y/+kVR1en39gMtD6Hrw8ACBhxt+C2G2zc1807NRj3pIw1GvQILa9Sm3VKdNSrHzGB+MBooUoRmmjjzZ/bRwHPWVj+sX+xkoX4P2zwUcfMgZ/f4GgSgHyEvS3oxqbSaZp0pidZyoCaLRgqVPiUgXQEm0HA5PDDULAthQOfxQveKB/ akraitma@aklap", "key_options": null, "keyfile": "/home/akraitma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akraitma", "validate_certs": true} 2021-02-12T17:50:27.066 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCgaz5AsLB+hPwnHjlf6vktlUQ8iWsKA+hViok+j+uCm0U/iho7O7Qd8E/3tp9yBx10Wkvuz1mmf6v2AO7Qa8+SE1QQPmA+4xLp4xXCqlUGFjnzm/7uQcQZtlWdfBHzQ9jVyPAgUpnGVUtW8PhW0txe8MEtecOO2C3Snx0yuTOx6O11jNvT938QjOeRQWw3X2YFUtMIOiwEhKtqstBIGXlMdaQKfvdlbA9MjyYy6VfK1x8lVcu8umUf/z56YmMkC2iO117/obhdu1XsU9SmjM51wHMOUCXSVqH8E3BWu0yohy4QmZDkdFihyYa+hoS9dOqGC91dFaQc46xrdmV4wqg1 jcaratza@MacBook-Pro\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDAV78mqEy/VkB2BthIE/m7iE5sbTjTRS0KuH7AB0KPvZBsDIw5cL8jKyxiaPC85+aOkBa4LrP0TUFBzN87Snt+PLMz2QHBOvlSTfUuRkHVo3vWJFzVuUSzzddkYds3hAxUriAHKv6CRSy1yFBxPQQHTSATUHKWqihMElbUCZrrrfH8A7J9vwixzCSLg4KbINjfa77rxefcZwugUqiYiVUWUSOb023f+QEaDm7OP+/uzz3SjF1Y0LMwa75yctttIfkRBKjY7uFCuZdKc3l2A/fdPxbFwlZmIox3LLngdTzHMVXGbVHqq9Ixo8FSAXiI1Buy2LTMoXMJwQYSvzsv7xNwujKVlUjoE9cOOIdzCC5AziwSiJB4+tNh73c5tRUzMMC98HLOhPJriSPvOt1pMEdWDX3/evcyxvRgJUKgYnmQG7n8v8O/CLyG0sAJDmGC3U9S/3fUDUjZIKWSsc85Q3xRaJwZJIS2VZwH0zG1Gmynih8PPq7VeDx7tKuNm4uTYT8= bigjust@desktop-jsho3l9.lan", "key_options": null, "keyfile": "/home/jcaratzas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcaratzas", "validate_certs": true} 2021-02-12T17:50:27.273 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDSc2rWtLtZM0CBWwaxhSfd0T++ZTX9c8kQrYT0rsQlJ6Hxj4c3+ypwh4iivdtFpyz4clxWkb/wimi48PvjNyZdudB9BeQUBeFfLiro3LnFEbKhM1RwRI89s34mRwmC0LLUZwcdX80zc6/fKuPzvBzR1PiY9tWRyIpYErQAHa2jEOE2MpFDHnlbXdfXFh5Tw3OmC1qBlQvgcrivA9lVaVlf+hfwmOQwXFIG3PCJLNlXDshCSUGAfYKBRvQaKS8sOf9rQaflPrwqJ8hDgVVQkpp69yVsGtahbBmjs6QS6C8EkGUFQmvrIcrQ8+5mRlRAmK2DMTP3cAcsYcMOJ+9KwP+H mgolub@zhuzha", "key_options": null, "keyfile": "/home/trociny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "trociny", "validate_certs": true} 2021-02-12T17:50:27.285 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:27.493 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANarZ+C62DPffRIhaCuCtXLChrfqwXgWqGHOLLkmPTtZ3/EE8ZiN+LY1ZcmP939AxXi8Oe+9LepwAdsoQKZctlC7XToyhjIk+rVSPyn3zl3BTC2AxfCKq2vqj65a1sc35Whrw7OaUa2fr5iHSazP6HuG01DhJ9l8HJsLKmiwWChZAAAAFQCNmunc7BNLWrqyrYwINTfscowkbQAAAIEAyYERcunZYJAZ9Eno18kM+ylq0jcQmnPqyIvssWXTttUZnaGQvbVxABeyr6ZKuUyZQfyetofquOGBKPBuzV/7posl5ShDuAcxbS1zhrmk9glTA1yFLvznDdWdLOWG06tjOc78G2AmldKHCdPsmgh2D7WyRr4QY4rcCvwKfZ6+eEYAAACBAJZrOlOlygfPZ4ZB0+EbdYACz1+L/iYNNAOBp51TRutNceLVrNJNOVdmCMLuF0A1go4kPSD/05aJOkOyW86gYLA5DXaW3961+kPInOFAe8OTyQYE7221ThYHRnCfVFjdknuCDomj69wk4SsR+ZPXvSXs1jSJWKhLfUSe9n4bmw7w anet2@secant.private.linuxbox.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5CMht/crEocrHqGjf0sxE9AhNcW3XBxwlniJwaBJLhekLQEQJoxAy2xUUmPYb2/e7rdmTBxzQipUaAGWVk1FHANac4cbgXzNT+9cnF7EIRkPa/x9jaKNiU0ozMhu/GOoeDXMrHIDgVGWPF3Ag/r17ORpYnuFTjsqoCWyh+mqskV+lmAp04rv2tvqouHtEs6gLebKNtB8c6twhrBKF62Ie+rSr3r7hQaqiZitHbg/isbU5RMBl6H48wHAVSK2/uTfMor 2021-02-12T17:50:27.495 INFO:teuthology.task.ansible.out:bnuEYRG8HvlnPRvQHscJYKiZSt/G7XqZ3x/7c4czEF+OpdQbAgg46vh4fNVMxZk9KHwEkOE06ROuwgAejJ mbenjamin@duel", "key_options": null, "keyfile": "/home/mbenjamin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mbenjamin", "validate_certs": true} 2021-02-12T17:50:27.700 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvG6+jM8aulc5CID/yE6AXPXczxq8tMgY+2sdo/ihWpgDZuFwGL4hwj8nyh5cTqZ2LEa4kRkUUIoH9avLCFN5pDBVvcj1/O06ZGnkq/hV6791UiPF5S5RGmz3MGxv0RDfjA0OLFav7u1s+ZJ6z2W1Tpy/W+g/fMYqbPc6rV+pm3Uo16+BWz/diZH8smqcty4chnnCy6gqOrcAxCZLpeyjX6Ydn3fnTLTAmhoQ9eUz7LjP4ZnFD5zL6jrSBlJ5WNTLevUzvS8eL7RuP8D3Qfn8ukIzPXG2vs594FhSc/HnvubKricvKMm0P0ki4Z0vM9VruQn7Tbeqdwpi1IfgyiAMX aemerson@work-yubi\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxUDk9jEQOSrRwe0WMGBPw28fzgOhvqMgcG90SvuUjR20Rx3KUfy9JMxPJPesjBQz0xgH5adOVg2JQeLrt3uSJdIFNpqox8BBrS1u/bWT2dorQjLnFEhrtA8Ae/h2kwLQ8w3yYNWB/PxDSuMRnyH4G8EWtVydzQcV/QnNZ9Y6eBcHaI5C2yVnFl7Fi4zBYc2eeL8o8gWqmM6DobBCvVrxD0pCNAsLhOW6IA140BNGT9F/hQZYje9RQRWbFKh/iKiUhqFYCzLcxFfYXkw2HZMJA2p/bLmuc8ZbgYaIiU6b90kpfDB37Xw0S6toIj9E8h+E3nkNnwraCQcbralhz/bdz aemerson@seasalt\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAv8/FJn7hz2WieoQFUGXS0bhwSwL1LCI9kkjWYNkWb8IJPrBetlkr5TsvHEKDPZGF5Sv8LsYA6njPnbOGyAHpYlnv/Gz4Hwq3hLgmXCs6JGWBQv8NS+pjwnG6Nh6JfFI0ANeg652Rk9Qt9uLYeKVKmHXXrVZAD4NMvJGfEvneQzuwwpylUmV80WAdaQw/H0EP4NScZhNhknwAs/+KPFZH+RFoUj/Cb3poIFjwyviIgyal5hjAEsoczCW3hAla/9IyC2VDS027WPiPFmBFaYK5lYYLStBQdRx+6amH76XScLFy0FEPLL5qiN0ClAptvh1oIURgipdXZwaxpf06AXG8qw== aemerson@electric", "key_options": null, "keyfile": "/home/aemerson/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aemer 2021-02-12T17:50:27.702 INFO:teuthology.task.ansible.out:son", "validate_certs": true} skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:27.712 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:27.919 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHk8c8wUTTYolOOmMlniNrsjfRyF9pCCnaEQ2DXR73P0hbFMme5Md5cF3PTtPIOHSNkWk/qg7s6gZD8LGyFmloO42RBgLsW1qM8RWTDbPy5EQdrjxL9byVzBBCDoQPIS2M+0XnhYa6CBj38Sw8XPlS41CebAJEOQeyLG2I6CAI0rTij6gsqDfALbIezd/eWvmbksKYM1Lb4Gvj+k/OL/xDZzIZHF/P+mK+trY45gsUHxKZamkeHyAT4f3kiNJA+UwzTWGhafy6UE7yGqhZjt9xOZSQGuwroL7EtAzntxayPRNxQPfGDV+fYYf0QBbTtHyJeikPUuw7aR1Z3KwuSmWD dis@zambezi", "key_options": null, "keyfile": "/home/dis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dis", "validate_certs": true} 2021-02-12T17:50:28.136 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "key": "ssh-dss AAAAB3NzaC1kc3MAAACBANaS5gZUnIp5UL/PM3xOrQiNVxfona7sD1XEXGYvNUyoojPWWAvdQRwdA/Ay8JsAjKeWUt0IwfWaPfbEoXU1TKAI+yyv5WzQUZAKmBFuYM0CUQnKrk72Dp6n8P2tnYen0ect22xEiW2+Y0JZ3wfV3IB3BQPoVCC9FQ+P3LHhYii/AAAAFQCgSceiZFeNItwJ2rJeQgVVTQKljwAAAIEA0Gd2VNCbbQ48vAF+m/i/CeQHJ9Dom9IOBN3v8XnSlOhtNzY1pJ0NRzrFQURCCeVQJaywr0et5FCXIDVqTuR4qfCu5XXU1KHpn/AvHstOd93Y0XYMwSXfNszOtpH7w9SEm7uBfKnLQNF843/MHZL4ZIdfxk73YjfMjrYFcvJwvM0AAACBAI9Wi2wG9Rrdbrgk8tqbnKMxxJCYt/Vakn5mQSc/WHqMUn14+Kb7oBtfTsfPFNK952FzU09ptNQvNWTPNMniI8l8lQi2B+Z28XUlFLL76uxJXWko99xFpf/d1kFBL0M7GRGmvvrMfeTCI+WIqh2jBVqbl4o8kjNx4p0VWyoiWeFJ gregf@pudgy\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAwSe7szyqMTqKK3D/JsXIa1PtPf1p8Njk0sMDSsJPA0gBGysmSgstke9fglOgeNIbNOBNDOY3vltuNCgUiQgVWOYhUaVpj+2vtS 2021-02-12T17:50:28.137 INFO:teuthology.task.ansible.out:oAWpXMQbNaWPGKhbxvJWMCLVckLENyoUZUBKLmTvRyKDjjq8CJ1TnfA2Xfo3a796EIOtD6essU3lAVIfIUILGD+HqptiHSB0spUJUVRxxlO2xw9rLYij1eUPRHrr9xWvDamr5JtKVtCLOL3M9QlltHieKlbJSCZq/EVxW9NAGA6IiqOhw2E1k59oGZUq4Qg+YLgYkB80Ms7nb1xpjQ8txvY+73xL/8i7Nqmr7ekKJfyTpRe03nCQ== gfarnum@mozart.thompsonian.net\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzqJQfdAKaggPDgSpk8P6QjLeyQYOz183fVSKnqAa5EeaCJozS1dn8z+iphswT3YL6wqIMqVHMfkSUcEeTBwZmvydaghOMJEIQSP1PCzH0V2XprELrEcBC3QTvaElPjjEYgBVUvEw8sL/RIQ7mgt9vxD0NcCDspsz35fmU222KzFxD41PZ5XzHQ3Lumwdax7mcpqrw8Aa5/tgcEqAIu54HYRf1Qmk6+ueMAvpwkX3eiSEtkVvuV1Kxpc3/0mOfJ7cn+eyPX3qUbo4oY76hIYseoUIwEaVE/pNW+pjwDSy4gSOcuxYDEGKzCwZPZeQtRQWzRt3YNafxi5xsByobGLlZw== gfarnum@GF-Macbook.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1ik3LbyKsPiGwdh+vGrKSYLvhMg7OkqMwDNYU5LqWBSotogxyGQJUuD80ZavxmmdXC4qScddHtWOJjtyX1CTSDUe7INpSgGPGnmSExVFpcYRnmxHAB+9M/c9VorUkQ8IOKLmRO/BdYma3Rl9NzhHI73yd4zJlx1K4NuQnhnj0Z4yXO+k44RkBWDdMicBB5lBP0hETaH2Iq1u5Dai3u6N5N9bSPXv2PGKvvU395K0Wts2CxMhOBc7LAixysNzNWLrnG1ghS4jPVG2TIHOElP5lIXUv1Wlkool5rTUWgnVY/7loWkDus2JryDM7hLGFXqgT7HHcpI4gSITqRhWxzsUF gregf@fedoragreg.localdomain", "key_options": null, "keyfile": "/home/gregf/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gregf", "validate_certs": true} 2021-02-12T17:50:28.342 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDgS5EfJyOxBkl/TtJikfb38rm6muQo7MwYufkLSBw9W2jFzt/ZMbVXyVsdZM4I6hWfZm1/q60UcER4yoA1xwYGOTmbFXrLUqqKxKTnCdyejuQ3miB7aiuisWO4vjb2+ppBwL9Mx775DPeQJehFyFK0qgh33tZO/9LWWYy7NOZdA9L2ei1PgMWa0VBeijcDR5GQSXzh0Yi+h/Ezu8jevu+NkW2IUJF6zPGjd9M9lAy7KG2dQjqqCX4XWmUS+MHJS6f9bsC5i39GWy29FvX/qFEArOMzuYRIFWE8iza95KSBxptb+Hw8PFBjQ1kTh6nC5ZV2X3ETcoj8fToTcgW3dTRF jdurgin@new-angeles", "key_options": null, "keyfile": "/home/joshd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joshd", "validate_certs": true} 2021-02-12T17:50:28.549 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCyOWnAdvzh5zR2ts9e8g0wG4wZ0uNr0YJUCncAOTJB0L8xkQU76jWzX3BRNU5Wk8VtLvpJAQ61P7UXyzgGWSO9FHZjD3RDrc2V9uh/lqRKmx1COtDvcart/JwzwHlsWdrYdcXvEaAiFrZJ6x1jJAjIL6tvMO7bqBz4YNruKpGWsQ6p5P/M5iIEPMj1oKanLoULXqReC6eYFTw3LNAXjberPY5GxYwaKS0HuzySRvSblShyA85jawTSlEbsOSnMqqGu5vGJiC415UFQQ5qKBtzrgAjlUgMbaqGTnTmBYn7L751gIWRetgWqwuHgfIe2ZGM3mmh0DX8E03DUbEt4UxyR dzafman@TrustyTahr\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDDtSwGzH7V5jzppggL0+nYlBSM1HqGjkInwxutra5iEyFoBuV73MNSBk3HrvIVGN13+PCdHf4Oy8ywbhC/4yKbxS1NG76d/GU8ijZFuBlEwxwHZpcThpSTuYrNx7FJVp8Yjl45DtD5s8FMI/0YZZhLlIqT6QCTxlTIPXDb6zefEWvAVziE0bmhRpe4B6AOLOGTB9fivQr5YzxhfaSS5Vu4U1x5gmosouSJ+aMFrQWLqa/eoCy8t9PnxdMjlHjHCZ7Qv1JfpaAPw3qOK30GrVeXR/hPN3aFodTuIJpzdynjk5HO6/VKW5+mWBO7pmQqn9LNQEjADXbLh7PG//nOONbH dzafman@ubuntu16", "key_options": null, "keyfile": "/home/davidz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "davidz", "validate_certs": true} 2021-02-12T17:50:28.756 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "key": "ecdsa-sha2-nistp521 AAAAE2VjZHNhLXNoYTItbmlzdHA1MjEAAAAIbmlzdHA1MjEAAACFBAHT7CHaceRLB5mJv9SYKog1wBXsavzH3G3WCBEvVyK76JIslDudZm2ycJv66r+VlGaD8sFI5Mb/KKFMKcKesLRX+wHqZg9MlkoJgm7WGoRQ5I0M1Vt7TL3uJkVCaGZHtO7BXqzTx53C1AaJ1HN+DG2MGy5mm6tHwL901UVqG4EzjBnjxA== christinameno@boxtop", "key_options": null, "keyfile": "/home/cmeno/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cmeno", "validate_certs": true} 2021-02-12T17:50:28.962 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDkIZyWnJXYv/z3EVeKHatjFUAdaomIMqzvFLrA9VskeM0Ctv8a83Wk3BbYvTFQWALjZjS2HVYzi5p6kTQ4qxXlcO7HxRw0j4otNauilvgIc4Lu5N4pUtYf0piEo+nrXz+WhELfmJ8t1Yx00uGEl9pOF04SOuXtpwwrCFAO6WMCcjDuYhoI2VBKO3W/OWcIuQ8ZY3j4iDdofFOUGyyoYVyOi/HILKjluYB+ues5+wBEkDuoi3MafCnIj8M0O5fU8/JoOTksOu779UN6blPiNdEgnwD00jA+RWYY6b12ZVkiF4rUVGCiiKKUwOOWkRhkA5OEcBgZ8cGWFgGfZM1GJA// ivancich@sleepy", "key_options": null, "keyfile": "/home/ivancich/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ivancich", "validate_certs": true} 2021-02-12T17:50:29.169 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsTZI+b02boZv8jVLHQoFlloD2WF/3xT+ZfZxzwddL7Q6eqo7j8i21LAbLAMWXYMyhiClrloDeHfCZ42gG5bqSThF8LoCw+GspF6SahTI0lmptGu69/OeHqqCpW1DnsWB3wWqfi90XbQJrpg3cdRaqOCmkGVJrqfFWzff8Zg9IuojIH/aO9drCTRGa1pHHCyEdW9ZNeYFwrMjfP+wJi/mC/ABAoXyDy0NWT0uIA+36abNg2/ptok90rdan0r72hjFEz/1hpupKWri+VXAepiyDMuSgJm9uJCXDFaNTn3jocdX+sbVtltThQ2OBxe1T60vRLbXx3lvhxxZ1fUkcP/FF wusui@ubuntu\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD2UzyfBk1nZ8SLCtW9UjorMeOo4u9AM/YL8ZI74GdprR+odERZR17/QfwT5vmLVNa51+CQt1HcGodlQYfR/8n/OUl/gJyVagBuU2zgmLPO3tOor5RBfms5MZ2VZ9Vm3+WrCO/vrAq/5HVyGbQsKiQHXkIRSL2NIgfKRTOVTNmuTSXtZNDgdWGIBmcOFQv+Im9CRopoI8ZtJLQj7xfxtGtUas7FbWTcf0o92+xvWuGWq5WQgmZ8MUIfvBLYLmicuVfsWjZ1YsC1PNhuhkwrW7MhaD71Ej3ll243tG1QD7eAmo+psIp8phlU5s/5FeA5y5OzDZ2T963oHGhq018TYO0b wusui@teuthology", "key_options": null, "keyfile": "/home/wusui/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "wusui", "validate_certs": true} 2021-02-12T17:50:29.377 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCw3MhMNDYjkiQADoC1mg9ga0Yj174FHIFt4RQIC8qK+6cB0N3CM5R4JwP8pR4tYkVFWmnYRScYK5VVdC873Ac2BzYxmse97zLJTbUnYWTdY+OFi1FUozsy3PwtiNHSd48Z/PBvkinQIey2QlMKaW3/yz9Of/Yfm4prxg9lBsd6/tOOFj3AsqIQ1OVhMfgeVEkdCL7AAbEYyMxePZ3UFrDwj05HDLgmFzx1xsNJFOuBwLfU/QNHoXJQrcVy0rZq9fB5rg74iz1JZs0uDWocQQK4Nd3Rv+wRGBuew2fmkrS2bQWXzMoSL+/i7pbf74LKcbWAE0wXEImqQeREHxq4ZZ5J yuriweinstein@Yuris-MacBook-Pro.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJUmq7btpBMjjODHXp1888xoBuX+rhiZg3ufuo69RgDxUMymJn3CPJ/5NGXoZ3SGAWKk0w0Dx6K/VO4y92mAV+cO81y8wZR4J3fPV3IbWRjN0X3IFoAvJi5n9v5hpG7z3RZCGmQnOZ99m1i9dge/3S8NpuPLIvYkSo9z00d+mNP6D8rGnqOmEBfEhUQXLeH7rbPwaMWPgB/hrpiPCyLR/+sGC5bq9PAsKnocqZ12cnvCtYxpnA7et6nL15ftEpUrKYAmRDxotuoUcsllP66GSszjbrgN6OBNDSqpGt/+RSJJwzZEGQ073gJuMveKy9tvD/PyEttjvVKL2KsBuXGSuL ubuntu@yw\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDmXWgRRes8bzekTH2fTcxdAAquFn1SjPhws1Z9 2021-02-12T17:50:29.378 INFO:teuthology.task.ansible.out:CkxNaqKe25XnzPehqrhg9etouUwvORQHdDJdTlmhTSYtn588Omq+1bvniab0KzAYgGZsWdPma/3E9qF7ChXIMik9Umip0drzYTaexCuC/Nu46brTxlnL0vaxnnVjDwmHC0CEJs4qN4e2wSlsA0uBDRWjxWFMdTUgSzII5xasHypUYZ0y3MNdbfY+SVF69DEDH8EOQMGYm8JQXNJFloQvmvLUFqT/63FsdpzRVTph4A8hvI1f8i6ULb7fPQN5ZS62wdXd/KvrBXrCvhOi7R63RM60mkMRXm1XajTGmCKiikfiqgTr yuriw@Yuris-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC41kRr6B/RC32DrQKRByalH832H6BpCQF9XdTvgYi4+LJt+kAuIGLDF4xm4uApTmWRrLspWm0Kksc0ger03uNx5qJqMEUnJvjxJnSDC0/XyxWapVxIviFkvv0BjbhVNOOaMU55ywrZ9hHgMdaFwuMuCOaB7f9Tm6IUXm2k82OXueeIG23Kziq+IQBysWOjvNUy/LLpl7l4+xOkhNUupUiQCqHvFH98VUdYWkHeyamafGVZmYbFLrOXgmviKo0u8LPxcT7psKLk0yFG7fG0DSak6DDdKJqb+G/NqMSPxRSAI4FbXQ6UOZb32Ozr1Sql9yfOXEYWZuatn7W5F8DjSv3p yuriw@yuriw-RH", "key_options": null, "keyfile": "/home/yuriw/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuriw", "validate_certs": true} 2021-02-12T17:50:29.583 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxrS30JmTZbnBd73s9PGZYxCkWH3u4En9Q0bYJan45W+j+y5vMzaNUVsp8We1tphqPQ9B0ygi80OxbHWu5gyv4g+M/JFtrnxv3QgBCYWeBWfh652Vbdlk1thl6btFnHUhVMSn5tkdnFxyVDJHIROoVXkHRaCNAUUy7tB60IhPAMomNcCGu9R8+opONFLXKTIfkYQ1cGbysr90DMs8ASEkvxdv40hIWhK4ufvL7CFFtgXuoC0c1WedBFOkr5LDegzL13xN97K9o5J8CAU0MUKk6H8KNKbQuNq1zANwymMFY599GIlxwai8iluJgfOObDRdCDmKM2f9RAEUXQM7oYoH7 tamil@tamil-VirtualBox\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvwiWVhNLeOLg72cpVIwbUYnmFh+Cqf/mE7Rc/Xci91on3GPrq++dC8t2GhTB5Zyq46EE27SmUkiZpHRnz7xwiJKM/tTgNEkffXDK3j0pA2E22iqV9FpfSA9Zdi1Puq7CPT7/rix4rF3SPvMtg/R1fkyornLFdzhJiMExCKJXGwf+QsKk4NVnr+nO3EEkKOoP1wio1bceZ53rCt9Tyu8NawAB2nL2fvtBPQDAriQ8bLX69XLbOfhFI6GJo815/j6CQxNmvA0l5ABJc8+A71jZmndzumFRgqBuhiCM5ohsGR0JGahUQjFEA4quNTx8EWTGRsqhAYszzpqJzuN+Xb6tJ tamil@macbookair\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEPzEyDGI7poyXVPG70vgQpvlU/lTt6Vvv5CoHWJoSrHklDdJEqekhCB87BX4DTrqbb5xoM7yfxle/z9UzgVwSUWqsPdZHA4EbrNFRKFxi/gc2YrygVSjumTyzf9WJJ5vCZvTQexOj7rjWVnJkjXHKMXZXIAQN/bJf6J21t+4A3XLrl30/4XXyq664NONUNjBHxHtlbS2LM6Th/hK4B23oj2BugX2j/50vWgd9H36bPr8+JGskOi6MH2ndZv5pYtLznFzQmsbixjaN8kCuyQdlRCOndPEFoHFhDla2MeB+h3XQLDXW7J1TR2URFSfcNq5ndDS4i0b+t75Fjhb7AtLT tamil@ubuntu-delldesktop\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6tJ9y3solDCzFdXcczO8GJEwMqMYXWU5kuMz0rr/US6F72j0spSNmwGemMOZE9s7KtHgior/Ufw8/vMRkRhoBY 2021-02-12T17:50:29.585 INFO:teuthology.task.ansible.out:B0C5KT9utGNqrtDdvH98h43XnKZ3K/yaHv/T+u2EYWNgHtOXMtgIVFnGwSD7RzF6TbwSfurprTV6ZUyyOxRcmHnkmwpbNwiwIoXB03k2PITdYCC3lEBEY0OC+vAW/E3+bFbV3RSEA0Cm31jmMyZyPeYiXiEnPXUIClsoDj34NwJTIgctwIsqM4eYUBg3pHliWYKEWDs55HZYoj3W/6Jw7MASzyAYFSmP8rQu8iS7tO1iEBRGOXFNxHirGQ1WqUn tmuthamizhan@MacBook-Air.local", "key_options": null, "keyfile": "/home/tamil/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tamil", "validate_certs": true} 2021-02-12T17:50:29.790 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDYGVqKZgLx2LJ49x0kGo/Z5r8MVefSUCYhhrK/DK8ht+YK1AqLXErjAedgpVz0WuwirRjh5YkiXGMYiOdkMUYWGk5nwCHEKnnCpne7Xh1OOqge/QGHlUigWVH8VIHBsZ+wTW25mqOrgDSGB8+Qk7Dr2o1sLGUHcx7vITDWOrer4KjF2+H5TQH/7irUVORu6fzeGKtGyOrTwQ+ZT1+fuhfznDULFGyvKGdcXxxNOg+21Fssk2nCNTlgRCGjHSrfelfuMT7QEt+9puWd7MWXYQg7xx8jkJGMpEQRFxJTxnfUlUXf3x2lc9Q/SvKfpVMtc0N+bk2F9+euyvRvjtn4LJNx jowilkin@jowilkin.wilkins", "key_options": null, "keyfile": "/home/jowilkin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jowilkin", "validate_certs": true} 2021-02-12T17:50:29.997 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDP/yVttBEs7mpVfJL+reXb/BfB7DsLrA5ArVqEJjySi3NOz3e/wE8KRWbHD1UdraAU3P8b96GdAQxGOs4n0muOMejmaXWZCP3HkRL+rcd+03ZI1lb6ga9XjBARZfb2h+kg5jTJp/Ll6mwAYbSXvQ4MuDQzwGRDX5Q5Xme7OZ4Wb+vBmmkihHydySyP9YXbWWsBw4jvZCFSKMGsUVy/HIt1coZx/PyCG/K2Kl0nrE5ux2QqfdTIpzYo4c3y6uMIHDvt29lRmk3Gnh2//U5eN8mmRRKJlkvE8DKy3VYsl0JsP9pTWUZEYgV0EX251WTO0akQ2bKuszqd9Ai04RWnXUu5 brad@dhcp-1-165.bne.redhat.com", "key_options": null, "keyfile": "/home/bhubbard/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bhubbard", "validate_certs": true} 2021-02-12T17:50:30.207 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAyRDKb+InDsnW4E/S12Pr6rK3wDfUG3j+iw/NsCWwN5xpJEA1/NhVlRzY8/jejc6ddmnNKWbm/UcYVrbo7yClRTGu45g0Qm9lmfvgDgi1k8mwUCy+edbQ/R8FB9cIHHCnUuWzRmE2XGaeDiSVQgiYqq3mOPY/hwe3G7jwFalYPsbmY1jNqpzODmKFMRd1bOwdHEOMTfh0owHWqhhXakU8em30lYraQVYH24FplH2NUBArxmTuctyfweHxx7yH3ZFcXFUpUMSLuVDWb1A7fMiL6eW6wiwh+dD+q5iFNvuioZITP2EjIsb9nS+eM9bOJZuZ+5pln2EsHee27LjunbwgWw== yehudasa@fatty\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDF3PxJYgI3tS7SG+RnUvxMxr0/XwBAio0oiP4AC+o8rUYYyPd/RU27TR1kphbuJhQWVtaZ6kFSBQeNQUxv6rb8hmMzvlP1FDEoGkjaOHOuYEc2DFkkEAfaR2AlNKn3pUQgLE00TLHctCgd57Cn2Ppt8xEi/vTuo0gX3xjWOL4N9QDElCXKNdxdyyJ/RNb/Mid+YxgKKjf8ppmhYsQ+Qqu4YwPzLkfK1Qb3YtR6Hbjb4jxC7C1Z13h0vUgHe7qSmUCYNo9eC+lEgNmmnLb2sXJaUUJM8MEex6v+4muWM9hV0cq/Wplrx8fhMWMa978sxH8yVkE3Svqc/K4ZWN59l3xT yehudasa@yehudasa-XPS-L401X\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDvplQk8xkSXF03uSJh3H4zyZHRMjvgUJBgGN27Kb7gAlIQv306wPKIprOwp+jDhEcPnEWBMRJMUJuBi0vEiqmDHt8mg8ZYcs+Gi7MbUgBokILdJuSG89KAgq7gBCuIhYbSg35pfvXvIZswiXvySqMrJNB495GYj/FrcWEBfQMuedMSkIfNzqPV9yR5YRh8fnSs3fjCzV96vzu7O4AzgwICbM6XmqVLXPax/32Hq0dz7sAawlQFCJYPHzVzjwmih46 2021-02-12T17:50:30.208 INFO:teuthology.task.ansible.out:TRJnXrxewsRhk9GwKfI4SWaFIRdOdadGFSLgdZ+2iGE7hTZ3sRgkLOj+4zKa+xTbtjjXfw5up7J3hAmrK15Tz yehuda@yehuda-940X3G\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfNxvAw6cb33jg36vw8kJu48/BttmFJ4BFeanYqwRgtWZrKpAAGHEgCiNeyIttMrI3t7Spcdyp6dy6n0P3WUFVgHWEYtGqXwXqDC9UXISkUJVVDZoEynpGC92RghhzH53+7jX1Ytpr30r/xb1ktryTPCoEJRqyrg2j1b167i6PV5tuJCA+77gW5LWBmMXJCprjnfnJ92Gp8ZewBB6gL1x+3RFkMxGsibL0Qw1CTKfyAHNYc/jos6YiL0b42Ha+NmKwvzZ1GgIleRMleisSOyKPUlPdR7UnmtEXa9bl3Gc/hc7Zt+v4iUypr2WL0X4Yl7rJNH/AAd/Z6jPkAX9nHYzh yehudasa@yehudasa-desktop", "key_options": null, "keyfile": "/home/yehudasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yehudasa", "validate_certs": true} 2021-02-12T17:50:30.414 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIDx37xbP0jYqopygG66LbSMRPoqpGUPzQZ3G8eqJCk2I dang@revan.aa.redhat.com-2015-07-17", "key_options": null, "keyfile": "/home/dang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dang", "validate_certs": true} 2021-02-12T17:50:30.621 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2YyDoeKdaCuJif79AxJAorqpUj33WHbHu/eIWpHju+iIF3f94K0qyEXgs4AwZin9pzfyd8N1aKNT8ZFPss6L0Xk68aO/6tdInq+1lTMMUC5v/LBqsIV2btPTFHrQWA0R10v05aZFvFaSqhHNsZaj98vQV7P+jiSs3yPWqCIWUTQ1VXnCeQnB4R5tob0gpMuonhkKq7vrJPhf6ge9Av+yjL7+8oP5LJqFJ5npDxkwZ0u3L0QQ9ftAT/e7Y7ESpwpgaI3Trl0EwAXFmiLjXEUA+FBbE7ko9kLqXqRjc1C0BdRG9sO0QKBGDBEx1W3w8rCPKgaDlVyqsBT1yvIjqqXDOQ== borix@dhcp-31-190.brq.redhat.com", "key_options": null, "keyfile": "/home/branto/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "branto", "validate_certs": true} 2021-02-12T17:50:30.828 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDO00mus2XPX/EfUZ/YsKI84TnWj5TbZP7Fs01EkoOTPgNoYysVdf0PwhAooz1qPrv+HUsUMpQT6uNYfCHyqx9rScR+XUlyQuFe7Sjp/dn3y+b1PKa6G4HX7f1eejR3ZaLZXA4R/8oucAXqQS5pDr3Y7prwkrp9cyk8uawBXoDcAlA1SMIPZPrtO1+iZw20NDhixsG7NE5jMFfw3q8xTckZEqBbteybci7X1v4lEYXCVlbSbUhyZ2Af8HEj09JBDOL2bI3yLvj9a5Oj/HtKubyApzZpA7rgeoii2UZQGuSRI1VJktp9YC0zGk5S843xYW92utbALKyQ6DdB2F5gc0DX xiaoxchen@LM-SHC-16501223", "key_options": null, "keyfile": "/home/xiaoxichen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiaoxichen", "validate_certs": true} 2021-02-12T17:50:31.036 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCz7e2G3Ziaus2yMviKh9s65e+yaqy1+ezs5IC6US90/8WumNAWqC4Yr3ckvCc120r3VwpJWcvaAWCF+4IZeO43lSNLadbCnJONed5YklcUuIJvjC1eDbZTMLJ7h0ijgNXgAFmFq0AV2NHuE2L2VKXk2hsIqDVGjZLUpi7W+ic07xnScEoXXDSk5mhJ6bKmFsUYCC91g2roYs9Ui5fGd+ey28SFOADI7osDLpLkiTF2OzGel93o5qobNZwKR0AxGsBIta17mvf3I3307bGnvugDXpNB0FaS1QZhIzMO4G4LZL9zahCvv4wm5GmuO4F0+Ac7dwFu3KZKtHQe3g6WHuAv ffilz@redhat.com", "key_options": null, "keyfile": "/home/ffilz/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ffilz", "validate_certs": true} 2021-02-12T17:50:31.243 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpGZK2cWUtE85Bt7fl+kSeiC0lM/EN+H2X3ePGq3LiD4g5R5n6nJsLaJaIZXdJ96G0JSnj07G8Xu1CkVbGDHuVg2O9tnxLhz6tzDwFYNFrQIpALlQwOq8OiwGDDyFkMXQ3JqCCWvyD0HLysOJQIBIlb/o92ZCSAGOyNX1JjrhXF+mlBk1AUW11UeZOeGZQ34MKHapH7T/DuexCz5gfZI/jY9Rg3hr4jb1zGttvAF2eeLJCjqD/Vi4mowYB/Kn/PJhNixeA2sli2i8r6yO/bFmA/PWnfSl+EqD4mUJWb5ATeJ3hyW8qVLOogY6XVu4Sd1VgL7l1RYzXuxJV3Rb1Yw2T joao@sepia\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDV0jbKpbmXW1bhRIOBIdVV1L73qjls7s5pxiW2EMPvZLYzxLBE92e3Vf8bSMWlm1Lyb7a+u5BR85+2c+h2F59sgU1Saek5ebPpFU5qBn8AzYH18iMVSPciByOzwS0NXwWnksckWxUkQgP9bZP/1ORQ7jWZEOHZ5mEQ0oMkP0bfG1cpFrNTxOhDmwbCiQ8dp6KeZaMg4hErALr5Eh/EESxKIkuwQzk9bDbtjPRDu0F2dE0NCjs0k0BXmDA80GksxuOacRxE95K6/er11qst/VqA7o6BfhxJMTXWMdj8NhzIUNjZ8LqLTG6P/XGR24sA2d2WBhlEKRMZNqeEjzmi1REr joao@tagide", "key_options": null, "keyfile": "/home/joao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "joao", "validate_certs": true} 2021-02-12T17:50:31.450 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHhYb/mGgHbEntAN2iPFTV1k8MYDs+jtsJDPE9MWtUZXksdeG8cEfXDBGhiBsprjELsW4+czJBqqESugFMevEysTVKIo5lc95bf1nApxfiRyMnAxb5wQ2OeLY62x26TWJhPu6HMajV+gb8CKjxDobf1Ix4ELbYlL77SOH1VVDtVZ6HiwtYngXAPMONCdI99cuwJlBpv5RjDoLwljOoxEXIh+1Fsvth+iymp0rIPuJx91DzYYgWZFfCLPHRgDy0H7YRdQWUzhrxTyzuT27vsNh5L8HgdowzOu9f2HwGzlTlxli94IJLMBY2359xsqjSswZzAv+BiPDAZMUwJpYvRvSt nhm@clusterfaq.org\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC1JAKymiRDyVar5P75cMycxz9u8VsLuNtf/0t1na4ofa4YmanKswPQQz1Am82pvyPqXlBbdIGVneyMrkEoFAk4nTtfmCl7WdtaxnH87Dr9BOLEp+lIPHy7jpS/dcVmKRPB1qCYZ5GNAZ9g+DrXdaeOJdUOpIK49BP1IK8Uw5eBWAx/ae6/12yz+d/6e9kBvO7OUamMSne6Ghe2pRoI8jFOdWMbQy3ofb5wSHZGPNDn0YTGbNWHlfEAcrFmGQSTwohca4SqCJ1NgKnYu4qPVmb3Gbpm+MpUPs8Gy++rQ8wVZ5xWZW/yReIbJCqqVY4vrQzVywQXRucIrB1VyvShuNW3 nhm@metropolis\nssh-dss AAAAB3NzaC1kc3MAAACBAM7BfQAinQmLlkTgt7IIg3nQyyaJRiWA4agfu/HncrcAmepGqPCwy451g1eLMD3AC4QptHnV80Y47YQTyX1VQixIQ6M9zHG6Bo7IXzyqFHFpqnDZN25DaMM4na5bf1CwC9RtR7VWubojatPylOpflkChUBuAGWnzwkz2cxl4Dc47AAAAFQD35l/Lg4eTqZ83eTgJ5lihzHS/awAAAIB/ZdMSE6M0zEvdNPsWY28bvQ0QXN7w1ncVXGachA1JOEFGYvsbHeoywLpSYoiPLV57/wZT2dUzG70u80IQGildZ1WgQNn7C9r2Z3ZUgKmRGyC+PpYEa6nv+U4CHdMViBRXxNrP0bj2CfQv 2021-02-12T17:50:31.451 INFO:teuthology.task.ansible.out:UxUxrHWtOu/da2W23cpSs6Wz8gtz2QAAAIEAvf4vx/32Sy+kbvqOCYz0wu0ItlECN0qOk3gAMvphYMI2sW2ez2+Tsua4eoPRXT10zFxiRTkHQd5FSIvNWXV2iqed6rpwTu5K8vJz2duMIyI+xWLHo4XudtZdNxggkf+RW9JvPOMLMp4sCzBXxpreHBsb1CynndKeYVrgMLybZ28= mark@Marks-MacBook-Air.local\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCfDOM2v4h6Dk4pMWPQxxHI5rhFtLjkme/4hAp6ki/kJ1XOSzBQdQmD2UPDKvfO9lKO7v12G2Wt+re+0giAcSkfej4+xqV53u7d2pTbMI5U3zohZt8AgESx22DAxj1wBZYuyGO2TRI7DRQMxC47E8EJgcmZBBnMKk1Rz8k71pTmg+CI2vjWx/2x2bup9Mqe4uMA8ubjg2qRlWEqPKIKcHZyrAPdIlpy2FSZ9SPa0mJpdl6aYcFNgonD8BO6A0u2RPO90KmEWrRWu69vRvbbTDiqE0CkJJlAlu3jeFVD8US9vymBHSI0BDFUbukuyG1tAujytqGBRLzxQDU9JOYMP45v nhm@mocha", "key_options": null, "keyfile": "/home/nhm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nhm", "validate_certs": true} 2021-02-12T17:50:31.656 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaetIvR7YgpCq142fBQtOYsJrOMW+Wxu1beNXPJ8qzT5MxD05sdKntVE0DPveUydaeI8/vHeNjXDKN8mt1eO7Lf/rDKHOW/Y/36bUfQAODru9sdNCEJfSPgq1y6+X1ISljvLBXOOl+nQXlx3n5OjThNQbmdthgCT2Fk4hEoK7gn4+ol6foTkTdUXIBam8LuPU6f1KIG6ae1oAVFWLeavAbvmxUTmVl04+hKJWSPnvZxWupaPuiHvsImaQ7GwBjKhzL1wSMly0+pa2zbVYZKkLBA7Fx6fzP7jcEieOOPM1YU0jR5JF+WVzeRFJ1V9n75yOw6S4ef/X/zwWx42cdfx3N nwatkins@martini\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDRivdSX6cbPrdMrjz1oYgn0hb8VQo+QU7wD+AcCQi/CH06ny+2QjSr7bfdpgQ81Wgwqt9+e3KcLD5qdt3tneC8sgSAR5QUQrWKhPejrqdV+3990k8k0KoKIOZ9MGFc1FBv5IpuT/2d535iHJb0ZEg8mFKqKDw71sBEoQT4WElatf5DeVC507hP7GnrhZgtTPcrVddOLkmmqTQtLE1zvo9Y/eel8O2JMImMd6mdVBIcFQ1alU1wV4jLdMpagoPelvVTVeNkGRnOgvvQac5RXtGSMP1R5PoGz+1FgRhDCtdTXRfOdtrlJPtUjYiGPpIoKIUymlvQGjUrrK3WiQIIkIfDtFFu48EN4drjgQpCEQ2D0NMKEmk74mjx3CJDOF1Nj/2Fa1kjECiz4EM7sR8dDG5cW5Y8XsDoomgugM+rsOH9kWFFClHUJmdlNGWLhJnu9UQHLYZjQw3zT0F+FXUsUxKJUVTUdxQx5HBO9QFsSe13NPDnX64+SQ2VVStt8oQAvFO+6260zbux0RVaJM5dTNwnIHD/uI3PeR1z2b+iKhyj/9VP2uBRpGydLw/etQu/lS2K30S+kZIDAp0sqwXH89bdq3AJBipcryrnW1Ov9K5Mfl1UlV95/MivrnOFISWN1F9XR1kuIIYdCkI7hq/FHZ9gd84AjTiERfxRv8VCIHBVEQ== nwatkins@redhat.com", "key_options": null, "keyfile": "/home/nwatkins/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nwatkins", "validate_certs": true} 2021-02-12T17:50:31.863 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDJD+2BHStkaMJ8RStIIz+yDJCGNT1wbwUnmuO71vNlFxzIyHGvCq3ktx2xTwZz/kj6ZayXYWIRH//BhK2/iK/v+u2WQMBURfMHbf0lDMmNfbHCDBjYvEYgFo4Jwvzq4WHus0AHh9TdVt+TpcxyGPhVbUJh8NZut+w4cRIKu2xGoZZGDeLgv9ijXvDPrCdBxtpvQnF99C0bCTdPM7MK1FSCQEaRmNUgzEj0H7R+gmWi0T4HUD8ZY7y0vWbYPqT3Z07My1u2BfRBYpXnv1655/b777TkSlzSasORhSTT0AkW25u90Rsj2JWr39+49/Ia9uOyFtZIirugEv2v7v/l0grN linuxkidd@zenbook", "key_options": null, "keyfile": "/home/mkidd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkidd", "validate_certs": true} 2021-02-12T17:50:32.071 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrRvNeBa+/GXxkSDmaim7U1wFpdBvgcK9/eP1I4fPS5xAWyxXI0s0ryX4Jwb7auhZdcqPjgUPQvIVVRridKJPz7l6jUlbIqdnZu4VHuaGG/DfIVxsSmw5aI5qI9Fg/s16JPwOUuzcLHsSSh8tFxY9L1yvs3ztQnKHW6nN5+zhPbW8UjmTNUksXXAeCM/OZn+IOOzzvfh0vVe/mghIW2JpOjHkZv4qwaWYcHg6+wWE4m8l84VIegDBiONknZXcQaRj82nHtfdnb2FSiV4+gKP18i90ai0iTKKXUkMJNFKp1T5MuF7UuJLA7l10FmgQJsZBS2Loa5oeqbHYeztNAfUxF JCLO@oh-mbp-jcl", "key_options": null, "keyfile": "/home/jlopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlopez", "validate_certs": true} 2021-02-12T17:50:32.267 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDFkUz8Wr2RPiY8+xf2hL4qMT+jA4mo1gmpIF8q7uVR03vX5NhzQVxVnXiyLCFv6susSMxpBafa7aLutqigXhDGJWUbsJII5yTnFYVer0QmudEsLmoDyvI1HCNk1IVY6aIESE50N3Ndx+agWasIU8TGKqJdDMPyiUtfrvsrGWi/91ZLX6aJSrFqBJtTpxDoNe4XL6QRPNv0wb8GF0az5graXkzVe3GkaQS9lyw8m2ougDvW1SLvoeE+lhUUGhJGkkCZKv1f+QNLhUHyQ3XkXqh4MrP3pPUKIuQ7nKiWDjCP6Db+ZcejjTRC19D2gLjhrw3Z2J7nn/loijlFzV5dp+BL haomaiwang@gmail.com", "key_options": null, "keyfile": "/home/haomaiwang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "haomaiwang", "validate_certs": true} 2021-02-12T17:50:32.475 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA296in+8Sr9slF8hrzAV3dNz0q94Q4FlFaMYw08II5PQm6HOpabGOGOwOud/PPeh3lMUHhs45N5WnbgdfQdzz9lpDoNBcqxeACdMicdECMHtYWf2GuFUdTAnPBFVl8wFW5JcziQYJePUcAUJRXGGfdMLMppAi8VEuhMw/z3jz89UN+zf51Y9eT3rgAVSCR+Zn47kite4OB2S1aVMEcZBE0fPnh60i+rpN26s80fh3wG2N9JrPBEvst6LbL0aEixrlRUtoFHZuWNCIjntVMANa+idpuWSxN2dzfcoxpGLF1TFzqx0moznp0PnDs7ZAFFS1/EkeSCgBflyhPV6csG3VNw== jdillaman@ceph1.example.com", "key_options": null, "keyfile": "/home/jdillaman/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jdillaman", "validate_certs": true} 2021-02-12T17:50:32.681 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAuTvkaUtEaLbLcYA9Zw3MoHI5ZPudKf6CAbAZdhp4RMVujimR61TT6LdRF20g6F7pH7GrI/1WP2bu7a3sZyCZ1nCJCOhrTTS+2wwy7oGo/0yY4l+Bw6i7LGmJgqpO5dDxwADMlaHUwJ7Rf1Q+Pd7NmggkAVTmaN6qUGffz9hZsj7ibD/Z8/GFtQsCEMloTLtnz4NB6Gxe54UXEKz9pvjpXBhs0xWsJtDKaPFrbKRKswuH45XPuwJhGr249ZYtwQqcgXSWhOUcPTwf5Y6fj27aL2dx566DxQzqzNbcTcO96MguIy8Ufnc0kIOQQw9oHJZtg82zwM6+2mWr9J/K94Gv9w== kefu@save.node2", "key_options": null, "keyfile": "/home/kchai/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchai", "validate_certs": true} 2021-02-12T17:50:32.888 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-12T17:50:33.095 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC/r/Xf58szqJ3wLdTE+mRK1Z11f/+VABmx6804NO77ncdC5ZvD3Kx6WP/oje01iwWEVXNAgEOaXp6j+U10aPvBMaS30qf/IhUcrXwO1YcGzZ0pV6UZMxcTuJw6w1VXy/YtgW5BlIfSm78zabztES9BqCgjYYM2rml3L1teDF4WU1QiI97iW0R57yc0t/gqXGpxqSFQ+iNYAl0Q1MGUgYltLvPrhm9WI2WhR2/GivdVMEM5lViNlegxneAZRb9YZMCG2kmSP+9Zd2NR367OrlgzmgeY/9u+KGXQiyfaO330s+iksxVwwE3fDl+KnqXuHSJNccJV7iXzZKej2QAire4X dfuller@Douglass-MacBook-Pro.local", "key_options": null, "keyfile": "/home/dfuller/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dfuller", "validate_certs": true} 2021-02-12T17:50:33.302 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIKepJjIovBTCkdVyvAdptljhw9bYcwtIqV/ojdbpeuJtxDo9UyrEkNoLeVpZgOcrbgrkOE7AEixo3qB9PbD8z5NRUleV7wipSh3QwrDmSylSVReJKOwYdxUKkzesZYBGIGYzAbTKqcVzPUWCeW314+xmv90XkO/+1vgbC5uoXfbfnaiDHQimBom8QV5Egr+EhhcuCqMEqaVSvjMr1T0e6IL9gUens6Mys6czI37go6QvmgEzkBbDJJ64PsiOOFHVNk4DJVJ1c7TRtsnpIHM1YVFjsH1AnAL6UufbgNSFf5dziUB/Bz815P1/jsOGpHkUKE37DbHU5xtLGl3MmimMt4ndcG6EggBGEYBwfEWPRAsZs3qngssrNH5W0CzbysLaoovzq5VqS8SdLMu5XrRLBEB0uWWzLq2bobYymk3t/oby5zrHJV78xAuLmldqjvlPZSdkgwWNv9YBLkteBVr2HhjtuashJ6amNPbA4kCGhXMpeRhfvfqs31CBEKbTV+A8ODWbWAoQxLCcqS24x0yMi73N0UzuqDTaJMcE8ODJvOxcLNaV688QDbrC/UehpX/Bp/DM7fLHj4YhkRpZxkkN5w2kb/bBbWya4DrffhHKqn9CH1z5ynj0237PhTwCXjtRn0JzpeRgN8OraFgR2H3uPc9XNqlkFxxOBK0lRngbxzw== owasserm@redhat.com", "key_options": null, "keyfile": "/home/owasserm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "owasserm", "validate_certs": true} 2021-02-12T17:50:33.540 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDVFy1sjGvvHwS7Q6E61ykH4dUGfxW9+a7Xo3JMpTBT33/E8bTrYF6ZHSdH+uxScisV9Vll222NJCLXyx2s9EOcv4L88vWDTZGCaUb3iA5dweLm7EYi1NXT/p+jWDA7MAF1FQGSAQafTHpNRxLcrVPgsLDy35weV4xj7ShEU+w7EBDb75jlJXqy0luJLKNBrLzz/wDV9Oe6y3APwQBhtO7vGrVzrWEF/LWLe7U9ad3DQjvLM7+H+QVpNy+nt7us73K8vHDk1KrPlCCjrcN+fSGPD/ILlxrDrDeqmmBCtPMa7PSSjCyAV/EW0fsmArcHrHXKeKWTKXG9sDL0KBqos1kB vasu@vasuSrv\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQConeaJ2kvUOMy+jeKvXZTw655L3KK0ZG0HONIySFrRNkGa/M/hYf8oMNVh9SEax7vLNme0kwk7d71EhgtMGNsvHtQOkshWxM9P0XPk4HgB/OL+2LQ/7M3SwD4vuXPbqvrQS63YMoxcHBue5Ol5hXmM+qSoZajpUrfB2woqbp7aKsCpoK4Cgj4XfLHfh0n239ae38DU6GoAcL0GzMsxMdQhWzBwOO+ZHfowohc1pfY0+xGIMW1OlMN1eO7kuK2gGS/YRVOnLw9uu/2NErIjBzCihNN0PYWur2JE+KmU1x6VMaky49cKmAlc5WUqZ+FpV3gvnb3cLjBClnCkdjIcLeRR vasukulkarni@ovpn-116-227.phx2.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRFyOO7qitrq6Hk2NSl28h4laRTxA2zezy1wwjTqs6UL+oZjNRYQjeWD3dDNPZrUmc9JyI3j1EN11cCsKAp8YXz3ItB46M0BPeJ4TXCpg36vxs7/twSDMq/EoXA5sRBbUJFf+bthcNemwG1WsJoxRVzMezbWBTHvz5i4Hbh4sL3PBbac3FhT/3b5Q4tKdO6bXGXmNLAoALyVZZMfoSz9CWzvrO+g8Houb9iJiaXHp293y+ND9Wicl9wHPckO1VcocZ+ldByujgX1cz6m+pxmf6AKY1 2021-02-12T17:50:33.541 INFO:teuthology.task.ansible.out:OMJj5PjmpN0HrgpV2QLVA+ba9qBiDwi1g8sRHXkOETMZ2jq/2a/mmTtvrsF vasu@teuthology", "key_options": null, "keyfile": "/home/vasu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vasu", "validate_certs": true} 2021-02-12T17:50:33.746 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDlOImwCV0AGu5L29ABaefV9Vqjv43WYJYFuLTxGQQdeQGcways6iNKGIX/nzf3UWkwgg25OVSpzB7zf6FjBx+HcQU+RJ7FLzeaz0s8ZxBXqWwtOYOvP9cXmAXA07hD5UAfxZKC1twBzL82bKJ2ZSmzfQmBjvpUOqRs7XwsQIClOW/ycY+jLgLRLFFDxzADsgnn6UEGlHeMETYocUbe89P3wbY5fHhqZmn6e5ncuAQMsiLRa9d+QhSdpD07Y/P65EzqRF4UOX+zpOc6gPlgHFir9pvf0bTbbBG/qKN9+KGqzZMRJGKenpHM6EMfLkf0QmrT4s4zd4ZnUQPq3HN3pul3 redhat@unused\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9n4ege98BNmvgUxogdcpptMILFfEisAQaiLUvSKpTsoy2gF7C6xSJXgDaziT5dW4ad2El4xmKQ/e6/e14Vx5i9pGbJz4NvL36Xmb3ent2PtY18i6d3CbnmGI8J7YH4YS9QvR7FC/KuvwLR6AdJ+uQShGhqG8kWJMjbqrtaxu8l1hq8ymBXw3O1Ht+xrZ/r3dEkAIVF3bMCSUnZpEbSsNoeiydEQl0ZCw+meXYmSw15vYVtKpYscZxqADqhSJJxIEHZA4E88dq6aYdV2AZ/s4cJK+tBl/ckBnclI3VGHSGn6oaHoO/fCHtKiENplDuKoLGQPO6xJ2Q/apHSGD7McY3 redhat@network\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDn/XgNfvqytLHgfBUdQIhJiWBlj+JrXqGZYvAZZJIWs32Ugm1/lCQTssGZrEj3+Qv81jLfkP6VQ8gXSWJkzUK4csxRxkZKXqcYLDE1csdmAFDOfWUFJwhMsCiiCyBiW9oapioLpvd40nw8k9qM4QuzaUhVonoGXMDDL1/dCgkbIjiQxxTJPHDEJFjtzul+VuhlhHqE6S1syoJG+UKo3Q1tOBCs8rwaPXEfCXFsyv+ZmcEv1+mpM8QRjdr1D1hfUfwb4CHtaa2AE6B5wL+zoY6+jsrTwZSwCfeIdgEZ8e3V2iXP5F7G6eXotu5r7mGYJClHZFLSfbNPN5J+EbRORl9/ wbhack3r@dhcp35-24.lab.eng.blr.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDHYkp1nuRSYSAl6me1xuoTU30DW2YpAR6EH1c2adUC9tNtTZbMnab11aitQLAtZb9SKsRgnoB5KpH+zJ0VNKCuJU227r9vZwcVUruSGwA6X7LOrSkK0gFqNRZkVoKhC9oGHSU0zegYBDxx8VLxSV3CE56b+KEosEo6iJe7TkXy3wXHK6am0uhQPEdSHtdUdD683CGIffdF1RFmAFnzTPn96lC9rLe1mIGO+jVQx4zIo3i33RFQLy++4DtQw6Lrgp26Xr85HmQekjgyiya0VkR9Ypy7fgF2ucBGegnOgZDGCCY4G2RuqiZyK 2021-02-12T17:50:33.748 INFO:teuthology.task.ansible.out:z2TZaOecDwrLkbYM7EGp8up5uLFRZsV shmohan@shmohan.remote.csb", "key_options": null, "keyfile": "/home/smohan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smohan", "validate_certs": true} 2021-02-12T17:50:33.953 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDZePxJmYerihzA0aQbMt7K7bctPEDcwPHiO8EVuJowp6zwhSmtDnYuFybKJsee4l8+rDKR6T5Ugdl5pLjSzQ+Ba00OSH7+qWC6shvVDYju5mIooJnc+kwx8RgapbGtDdl/yYskBb1KD5g/3+i/2aM+Igk+7nQR7Cny5+WnUEvHd7PHBbUXQSuu+mlUlnXCVMM8g8qlc1I3WQL5eQoIqGjSWRgZUIsn3P5oryYBzbuG9jqLwbaohJvZRJqkimQDe7zCFk6bB3z39pyOWUN+sldRvyL8FPyEmqog4IwpjcYyCuCnPhnb6fNq9yB65RUIde82Kkzx3LZrYOXTrdHgD/lr abhi@trusty", "key_options": null, "keyfile": "/home/abhi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "abhi", "validate_certs": true} 2021-02-12T17:50:34.160 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDcsBe6CzgWh64iZAH8zbU6ra4yEiUBSsiUn0AHzyQurvjeRpL2Mut6B1WEi0kJG7BAM6b13JrhnD9BW9yIMIUIPpVTa2o4zaXP90+bvjGrgDeJbxv3T9/TGh13k+WWyampxm2lo90/+Paql/HObQyyIv7Wan0DOdIe2/Ue4ooWH0tu3kj+3unJTuw82oEDEn+9tGs0oleG2c8XN0gQyVl1DzR2Gqf9PCzXK7hInC3niY2huoSAZzVFEgsniKEWqTnVWYi49X2SsJhUyHCj+mqrTG3YnW30nha3EXOWIbQGblXVoTIOc+gNreqdGByOMICKSnQmPSG4DAl3c1Y4h4A71T3Q043IUv7v7S9tAEKq2h6oJ9I/a5z0u4kaL/+B7Cw7l4bitTdcZ0rcqSvOkKUNNgxujbucMSSc4Tsh5L0762xaziD9nQZl5bteb3VtsMAueqhu7L3sJy69+HGg7oF68t5TguErXfI2UNVGOn3eRj+zN/zSwLUXMoROlpM5yHXSzHvpsbgZiHBImM2CeG4qaPtqzFw7VH6NWJv4ypb0htOMWsbXmLshNTca1KXrrBcVozSJC8wKgEOxu4spk26G8x1IVTsi2ypyYX+irtAqZrrpIPdQ0Ss9NjKPhFNXAxJgLHj6TlH8UNh1gRfN/QTyzpFSe4EH6uOlnWuxcuM8ZQ== cbodley@redhat.com", "key_options": null, "keyfile": "/home/cbodley/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cbodley", "validate_certs": true} 2021-02-12T17:50:34.367 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDrtHyRuOSEXAM6mwL4qoM+NM4hpIKhaRS+/Y6RDoFKszkXFw/jC9ZYh7q39rQpWQBKHVe4RyPN9yyGWydFwaR/BjZi14ippmYOqVG4f7b+k39NFbMJRcSxYC+NvkqxK7+zpsnIWC+RdJ+blZi+zCbaaAYcMDSvBZW19FsJSLrO7hh4R1cuzgV+Xrmwdc1/VH+HqPMu3tHGv7N0SfsklWI8Inr0aiL83QqqgKPcVxljSWZGYvvFyTQOrkNQeaLURJNER9WIIac9o57WOINw8kR+ojebcOUBsrxZH9j+gMjsJ752Rph03wdMUpFcnS6yfvI4Ial7x/sGms4FE+wL/vVF mdw@pika", "key_options": null, "keyfile": "/home/mwatts/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mwatts", "validate_certs": true} 2021-02-12T17:50:34.574 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": nu 2021-02-12T17:50:34.576 INFO:teuthology.task.ansible.out:ll, "keyfile": "/home/prsrivas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-12T17:50:34.791 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQD3vdP9QHyNFUNPOdplezEbdYsQfcC91ncATI6Fl/xHifmUbcvjIRqZHkCY6FNWUpMeg9S3+wlHJtzjVqJJ/MlQML3EtwnzQkeWv8bjtuZ07OqULYIv0zEx63XUfSnrdFleJ8+KDSpn7krkYE44q85B88IL+ZBeq5eCXFJ1TtmsDcvsN8WqZNWMdUGqgH3Bb4D8+YKrEfGxoievyxZ1A1DnBHan2l8foTKYggRrnuSbTb59cFpxY3R7Arc5Cn9kDJT1nDVBA5SSGCfRMiNyBOUzQVYe/UjKTaCpngglXC3wYnjoGvGcpksHLQCxyuJPB6djC1tdF+3Jehdf8y8+2TBzWXiiPq2XwS58sER5GcLLLQv26pPL8hgu6CdI0sHxgcgQBgpjoLwUMHmv13CnN/AAcZY4RRDK2X5zd6wFSEhuuCoUqh/m7K219IxI6fVkLaApC851D/ua/pj9OMjiOaJ+D13cPrgfqxva4eK8FQZvqQm7aUK/TT216QNYmn7hkzNjMTkjv/L5yYQ5Ym92AHy44wHzLPrtDB5mk94vHOjG+KlAt9ofuLncMSAXWe+fOqxuIQ6VfbL2LUv0EGYhHe4XOVj98q5utsVQxQbXAZzNQaq9FWXs2A4xcVP7aWj0Fooi17LN/W0ng4Iijir5hyLr7k7M1G6resgiY7p83biNow== pdonnell@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQCpgLVt7Zphmf6nqD+9BC0N91jdmaY6ZpvrYWlzbhxXITQ6juIEtegnlZ+8ohdeEfHFBiyu1xOSQSM1X1ugvpPQE94UqQg7JVb0tuoUCNJHYnIuv8T4YgyP0/vEV48k2G0BBWlP+IJ5f1iDvZuLHWLR5JSgGqh5ELaaHr7Cbyb6JlImldb7SDIoPHMCZ+n6dLO8l5Zy7qWelw6aeA770xmPucBr6t/BVns2g77XgJz6zFd99yEaDy27i8jfzL7WqR9i93yVE9EsL7tuOjg6G9KUzxBF0bg9iVsJXypvcBuG0JwQ+SDKqYw/cN4iLy5qBa6Siqrfpc/LKU21VNmJ0e5eXNgKKbMBpBb8GVmkoLaSRcXw7LclWopjNDA+meUhDAwU7b97N7cqN8BYhBKQyiShXFHMKLuNo2fSuTRt/icfb9VisKNGBdvwgV0oOlrqbXMzZKmj9amb9b+3JDBTGB7ioCsj5x/Evf9srXRrQ9kikIr1aBbVXIWir/lasNWVydcmUYsNi8L7W0vXArLwp+GgOXJFeZUiD3z9LG4CouRxMSwoj3lgWeL2ZR8feH4D/SP/+QmBdU6+aG97au9qreeXi5X8QTR3LND0DwwEKczX+3s+Fa950Oieom/gXC57PVKVh90KVoAGw3HzJTlZrauOtLkYbTUAX6woOMhAtJ5oWuDDT4UrHQj37jRrrl2dETLwKwUJN2+2L07vnnOZ4ABcGWLsU6d3bXeUBlw3pdsYqPebDKUEpjL2kUOw+zmfjE 2021-02-12T17:50:34.793 INFO:teuthology.task.ansible.out:fdVx9BfrvExvDX+3MkxccCJ8FY7od1s5fGEFHsXnIiMnLO/HZb/JN+Aoe+/ExadIcYwJnWcCJyly2/532kGFUlzOliCwPxgJpjWCKQAv6LjWmxoPXiA4JI0bp6y7lMEjmV2Bp3mwphlsW/r8Ie5hg1UPh/ttHRp/VZgcV8e6nNmP/qc6k9b2DWTNFqtWqm1hXLOwq08HzUEeH9mTqvenyAbv2RFMOS18Y2K+4BKUzHwZujrz9qEh/kqtpFScEIXkOJ+HZz3mI6XfjWFIPS487yRGfUvW3+nSWhvHc8vJY21bFRKdjD84qCVlZ2BfOCGUZSKftVItEiqXMTRs3RZd5epfCVZbJuA0hG16jUseeHqqZqDtlbLkzkxtS6v3iGRnPh/Ke/94FHHCzvYY8cmMYAVaM6p/G3edWftbaEtN70Fo7VKxPrpPzuSRZ7Vxmnb+Zad95DxxAC/vtMy/vkBVIP6dyn4HGxqa1oECFMe4oXdhnxZQoPzvWxb8upcXeAeOMMbcIlMq5a+dCBPGFNKhNhrb+9uAEDHwjE7rZlMJVSvt9v1S4hll6jiYIap+7n4reSKrS45aO9U0xiKXJJDohh pdonnell@icewind\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQC8MdUJzIvTCn0fM+hv1HAO/Lwjy/C/V7stQRWGIxyrRHbniyX8iuaU3j7YNFglSQPJq+YRM9UnGauM9/0J9tHKLsMp+zRQYrtxU/ThiTUN8TRJY11mGUol4yLsqkTtm6oq5sl6o+LfuA4hzaYc3XLrFHRJuCUgbtPG71Q0C/y3U+itrDK4wXUxBa4YzZzcIMg+c88VXwoouPbiL0gW5PCXzMaNtKlOqBUKlE935MLn70wIEfN/EwAAbss0ga1moqeqmi9Nlvm20DNIVFmkAGtYoq6CHXtL/uehzai5Sueop8Vmd8+1U8gDTo6JpiagdD8ZNf0zAbRx587KAT5ZMnahJpO276j7RNa85ZFkNkEnBLqBHRkCkTCYq1GehYyRaR69uuKyczFiIoTfAQgPh7VlnpT9525b/4nFsBDOkWxxWAIJR61YiKemPV/n/e9SwhvVsMDb2B/RBriwR7Mi2SZ/lBHGB+z0OA+EE2rGZy4LFVOfUczqIdpltlNotXan4M4F361i+bX19KE+f2RLySn3+GiFv0CdLwslS/y4xysz+FWxw6B0Y/EkpiDU3CdWM61+THGGm/DVhfL/Pvca7k8PonHFYw/x5w1nBcADpkM9JF+m5eOnp+weTEzl58sS+3GnoAuL8YFJwMW1RmE50xM0o5E3SDc8XftUvaSZz8mhII3uwDDukbELfmkLbnIeipQKyjO+GEx0JBp4Itz82qGuao0Wi32U9wHzhSWnX4GQ0pKZnq9frSlyCNwHu/iMiKF8sA2bX3dvplFJ9nKWPefcU8bdl/upr5hYrd2GttjUSkWZa+UuF264yx3ztcN7jotRhEmmdEcHVzKQ26/NyRI4/9DQpDKY7zw4O2wFkXWg5L9kodXwWzgy4cyOr4dMD9LBVLOpHc0CAji5/+lfuYQbu1mwVdfOzCXDqByjt7zCbeN7KuUh403sJ8PTQVp642xA9i4AkPv5g+fqi5C0++1DbZP0lmLfKtTswHhoN+kAqZu2N4Mu6mNVNKyZgW5h1pZzNSGWYcZttyYmMCHhPketReL0LPJ+VUsqQCVohuta3aaSe44InA28GZsXzMj5B0el/84bnE7cN4b+rfGD25NT73JEt3RgPa9W9jqT1hwd/nf53BDKpp7DlHjR77WSWT4h4F7bhbM1RsGSqAfZjCUJT6Rq+IuM2XcrLDY8zIeArkqaZLXMzfGj6KJ7cQes05P5LYfY4osXiq68Gs/7gzF31PQRRGOYAoqMN7752sn0ZMcpnUj3mJutQcncRUN4+naTMjU36WWD7it0YuQBnb2IJ4iKjYLInw8pr2zKgWtgr1TSCM4xKp/v1/ZMmbBmS+d 2021-02-12T17:50:34.794 INFO:teuthology.task.ansible.out:TzNC7Mpkt8vdCNjzZ5KZ5 pdonnell@aglarond", "key_options": null, "keyfile": "/home/pdonnell/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdonnell", "validate_certs": true} 2021-02-12T17:50:34.998 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDKzhyWb3PmU88m8BbgpEuBzYM6+x28z6aZwHSesiyfXMfF4Y7IZjtKaTxH0ph/O9GQCPhRCVhRd8TlgKWJGhts1E+VkQfHfT18CgmrPWzj3gifc2FJhL97vB1Pd3jksGnOPAikgYPHP46SVMTtyvXCkOUuo7E29WrkdpjawcqM1rfpw63OnzHf7Ar7Sqh/othB7ajyVQ92t+XWSEqj7K+NRfTaq5L5RBWbwarM885o8xdI5mimOZ0QjzKOhFDORrI+HEVlkutyB8E20nZgo7TUy+kYeLVhEqb3A6Pi2QbA9GFLT69kA/H9nebhj7Cuim9yrx4OKeZuIncG0UxSQF3QmWflAZHBs8L/jniTz4DJ5id5jR29tvGK2LeddCEMB1L2cHwWHlG0aX5jaNEJe+nG6H5Tcg3+M1RPM8twBkRQmS3UzAyX/MTH1kktyFUHn8CdBXe0WTuoFSvXLB6BCHWzDYZOt0tIV/xMtBwhy+XGYA4oZSsDU5m7VlSVLBGL8DbRX69Z8AGey2y3FoFu/roY6NmDhTxUedJ2ZfoKSwdZD6jjatdO1621nonBnJxrDgaXCWzT7HtnWcc3GwxZoBrqkvgpsx88f4lNi22OpYVPV6A4hECe04u6OJnHSWVk3ft3p5PaLNF8aEXbiIgzZKAwX3LxpvEDFH/t1sdihhpmjw== jlayton@synchrony.poochiereds.net", "key_options": null, "keyfile": "/home/jlayton/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jlayton", "validate_certs": true} 2021-02-12T17:50:35.205 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCijPApnekNl/dsY+fleKYstEzNkKflBEcCoKtjItUCDJCVLVFOsDBrV94EGvDWpr13aj9sSwpUzRbeu/vOW8ZT853HH6Dqg2+eoYwG6vDRprZb93nw6l4vtU59/M0yKkyi9CPp2+YUo31cCFzYWpq8bv2j1xBruDSUCUdn9nJrHiLFqA4QEwSpw+uVCjSKzQL+LjlKmE8Plfd+8mNCQeY/OFsjxAztwN+4M50SZCnOVCIwEXoYRNd82JOXMOaqZJazSWTO8Tml6+Nu0hyEcvHkHxWRDSM+6LmXNnjlVUPyDN2h1/VAYATpOZ2VhmRarDoftCp9Wzi7eQdL/CGiMyuD rzarzyns@redhat.com", "key_options": null, "keyfile": "/home/rzarzynski/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rzarzynski", "validate_certs": true} 2021-02-12T17:50:35.412 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdGthG4qx90VQ8mJ7TI/IFFzptkSqBHYZsFIVu06IQgHbnFBvjao6heGt4UlkWJkegPCDWpKVAe7wFuXDpPlj/VssRHB5LBnYHKGfkNAzMNiMjpEZsTdy+9w39CCujipz0AlOSfrUdANBZ11AnG9VgQSPEukX5487juob55lFM6J3x+v4opBF3rSlux3+thbr8V0w4oCHMD9/aHvVjFoBnxo4oirR4jc/jgC2fQCMiAJhFKJGhjkKFiNHxcB3npBuRnpy+59BcScOAzSXdJ7DGnwxQLjymnhqsAcWinjjJhOjaq6NaoeKv5I7zQLaVGaQdIyuuAK1Dnb7trpfQU7mR rdias@rdias-suse-laptop", "key_options": null, "keyfile": "/home/rdias/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rdias", "validate_certs": true} 2021-02-12T17:50:35.619 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEEjb7yrcY6I6XbMU0hCps1NqBuYmA6HDmMcS/l3i5wJRBitIXYVlv6pefUG9gdBYR5NneccLfVFURu+DC63MJa2ELJvVXsQ6YUOX39bJUT9KlZs8mH+nlTD7tlPmP17Ae3k8sT0H42Q+/5+b5YvdAU+03aPhzntmy38SyvLKrvowRk63wHgcE9p23LY9tY3+HSJu86cb/IdePk1GLjnrMeT9aM+5j2eETrmBjMArP+/2UIsT5eb8MBkQzJ299SCC2gfV/qIVkorFl9xdejctAwNwdT0SPTMxCHxaiK3wZ0e/5HHTbSgR5YlxOlKYq93XBVYIaFKPCePk1At1a3Pjh cephalopod@octopus\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDcrTZoKO5e5xs2PB2UuAruvGlJESTHNAtqipbC8yJtRg8WGVDLt//rGu36QNSbpbJYMZgqgKsXBYolJB1ckqLYLv5j8RAoVwbW4QqKaV95y7x0bbXs/YAPnRBhEO1nfpT8GHQusUZvTbz38FDaQwstsYoncmsyZwBrqus9ycFavpY4wfpKoWtzeiK5Wjq7ervThOs98ZJKXJXOOxeI7mDakE3Jv7q2tBNu/rU1kzxIl65kWURzgxox6EmIZXQK3FzKt0jjdKtApVYSlx4ODLPI4iyxOv+MbRe3Nnj6FDhfz8/FeP/U5wWxdYx2WslzZCOI1BJ0VDLu3xEpLlu4vPzp vshankar@h3ckers-pride", "key_options": null, "keyfile": "/home/vshankar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vshankar", "validate_certs": true} 2021-02-12T17:50:35.827 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+CtnvyVRSkIvD15O/q7/vQmD4oE8q7Mo6oT3kqTXaEaQ7iDayC2biUC0PC4OztPlNJxPUcKK5DfWw4F79OKeheqUbZRhsCC3Ge2JGoXHmsf0kBYdioMGiAxm+f8M/v1KESSSxvEnU7o+oi6VtFchh4Dl/WZ54rqZc/oQrlGNTHulFAuyoIAlKkCe3N0GdHN54PB+26QbLXvOuqpWmjVFbcjgRC62mkZl3LuCBhviXS4Rl5/ZE0QIHubCwV5XVrsqvscLS64bi06W8RN2vL56Vn3N9TGV4d2dlbE1PaBBESoTQ7Q0iUN32OAZLo7nF/1fiPY3Xc8r35juya5k+KR9z adam@TP50", "key_options": null, "keyfile": "/home/akupczyk/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "akupczyk", "validate_certs": true} 2021-02-12T17:50:36.033 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDeUJcLeyfqIk3ErPoFqh9kOnvUPsk/2hfaYNC2wZLMOSTIs3zXM8xjki8eN6vle0dmmeeAdwF+e2hJ3tCFKsEUQRvMc7ugIBA7L8/DboShYu7ItyQ3QRvy9pi6jnA/tlHA1LHalvyhMw0yZ4kvPbrfNMhZzOKzvL6jX166rqLA5AuiIfdYMBpE5rYq11GVjm0uQKw4UJpBMooNJeovEfQlNBTSV+IJSTiGBpBD7SgsuW3USMeEF3fEgQOcnvYdv5Fbp/crloyLWYc0HTidkLxpjP7vqY7E4BuBJug0zIB7voR3MYq6rAeuWjByPvq6coDlUjnw/QlrvarCbjjSaHdaRXLj/Dj+QqJFviESBr64ZynhHL21Bl6rexW7AwYyPa73Erbus3dN0aVQQ/ApcDTGHT7KRlDQ2YYg05YM4swpVVnrIdV58FzLWkll+ypCK/+kdtk5eRSe/KzZA3s7z4d4WLgolFWs8Fky8Rx1xDGewvMJgFUWVUbcGC13RocbNSxXG80cbIx8L/sclX4lDfGouu2sUXhP19vq1rJ7gZa6XTPohXjPbFS+En2cdlGdKXhppdvjT4QWKNzgQoF7qgWHMqJnMFJ6beT/2zP34KoM3t0ulee04z88VSn8Th8aqvLlGbdrt5LrE90escSaIAR1vNKZ1hCFwGieSKoRtYJTCw== nojha@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC6NWn9EzQJoeXkUbaNclYcR9Gga7HFhhXMGxdo/9ypYwlEzQTA6FnEreG3s1EmlQ4hofuc0kMhrjh0YeJZMrETWMoA0SmoX+6iR26UWVtOTk7i1v2TqkTY7llmQneHqQgmZEBXK0jF1HVP5kHY5dhagtcpfsGY5dc0eGUjfqAyh/xrCxciOE/JQ03tDcKIc5eOQIqv46JqnUS90Zr/Cov2qGKKHTa/VO1a2viOmDC/uHi5xXvII1Kh9rHp1oZ2Wm3fiGgT8xZaTJFFhWpdq3AdKfkEaV93yQcCdaUsGBHwDHcPOhxpBZpqZvcCR3I/tigDav4QViB1PRN7JnTIWCIznfrQK3r2Frl6BVFDvydwgRVS43ctEKPOJj8ELMUWX+fpjsKrA9Kn7nUU35ti7qIWKg1Cnz3lzVA29TsEg10E+nT4SVKZtRSWw/2AX0bsfxfSY30yjw5PP+bjnVh/bOOkqR2UkPBj4eeOKMD8nV1an79xP0eH+t36INaJhq6qwxE= nojha@nojha-mac", "key_options": null, "keyfile": "/home 2021-02-12T17:50:36.035 INFO:teuthology.task.ansible.out:/nojha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nojha", "validate_certs": true} 2021-02-12T17:50:36.241 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+c7zV5QmZb0WsgA2UZogoDNaoHcngRVLA8+BYk/U10ZlvCckJSfIXWMlBnqguZgdm/jBwsw5mGVBgHxUSqFNybwIlU00kyPZU6VbyVP+ZNVUf0XIE1NlemldgPBxxcOzSHkHhWrUqA3IKiCllcQVDQzutfUnq+jDT6NkVn4/cyNzNlargt7FR2L8gvskh62jLTBbhEuHMBp3jLCvaHJOk708gXm9qZ5iLOGV0pu7xdg58KE0p8ILClQBTbvoqPgZE9LA3qaOk1NR1n22kuW8JRHaKfD2Hn0Yu/MfKct783Bxb3mxWBKXxmM4otdmsR/R14NH/hIPce9wSkOOnQAC9 ifed@37526", "key_options": null, "keyfile": "/home/ifed01/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ifed01", "validate_certs": true} 2021-02-12T17:50:36.449 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCnOq6OHTRJuv7n0GZygBue/AE2q47pFM3KyrvmZxPRNmwtXn2r3ngHQQyc6DLZdFIv9Yn0VWu1KB/ryCdN3Xy7HAxBfHdKjtQ4vvfsWUbQF2JjhEjrsl+usgpqK1Tz6yuL4uAtFgLTN9o6gcH2bsQ15w+rEHCiYVzaGp7Cj+y+/VPBRIEHETSUZxwqiDH57uFTHxJrBIn6yhCorbn5QOjfmHUGmkxl3Geph++LbuQeha5F/DHaylMulsYZZBDEhIG8G2coDmVz8Cp1gw0kBWv63JHKnXaJboGXty2Zj36O/wdZKTE+7cV6JhxfJczhZBTb2cImIhspGwdHfnFgBoXz myoungwon@fg2server", "key_options": null, "keyfile": "/home/myoungwon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "myoungwon", "validate_certs": true} 2021-02-12T17:50:36.666 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAAEAQDZPRW12u7LvaiynB87ad5h/JFtYIVjkKIOym9Y5CAbvatHhZ9fB1WAnYIn2HqiSSPUXT8sFy8w0TmM+aH0/ZGcfXoe0CdTWLtUbXKNlPI5h7TO9rHXESP2iRBpnKs3vK8WnMY0/D5BeESrPfwVScucmzaDrjUqjdZKR/aEeqVvDpQsdaCaoqNg23t4nhQ54BEvK8O+14nQ26Rtmm40QHO1ze+0UK+7nUMMFwxvd1EKjVp21EgYAFEbyQ5Kbbl+fmS/11AGy/yTai0c4qd7ohBzZvx8+sY+c1novG9Q7LhtyxuuYwzYIrGBCkKH5+iC7nt2rGBP+mnTfIgdbJUiXX3cdrMESwOSZy6x6zgbXXME+yfM0myhomKmUkpeZAQInX2KYrnEuPD9kteyK9tZcG6gJSaiUdvlab49Alx5lM2hc62M4s4WKaIgroCoApd8FtNAlTQsdYuycntgNrbPy0LXQ5hw8K8hkvc6RbxISdj/RXWguQSVbJWdS9Pf5NwD3Tq7dsOghJWhx4vBkhW3IECVrPQWVyAmgQ9oj1PoL7/2LZ6h9P8F0KC1qmhm+pWdEPLugOlU6lzQX7RJcukYlsINThM1RGwbPE4h9XcMyHAnODiDF63TDfQ01oKbVXj9XpOGY5eP0EQxKJYjCfWW0Ot6nK2ZCt8OYvoRkWMGnLv11Nub3NNOgh5TB/4nDvEU1cIb/Ikb4BRzVIEbast/pqUJmXe+GQ+ho7mp8y8a+lIxu54BR+VN6isjwODCVt1DYohgixEoul7p5pXvTM+JUY2GD/ddTGwPvVAN66sWnT9bjZZunQwTHqasFCIHNyPhnEnYIIqCo9O1I/D8zRQG3SYNxT6ED9GPARrE/H0TvdOmKaT3e0F2GRJRSToNxol9dpJovUziojOhXyBUjlTqnMY2RVilX90MnzgqR3zGO0aSruTGyNT2Faymqxn6T/50mSdJ8EyWH23BaaXUUW/z3Cjov+yCQjLY48SOWwlxKXw3hGz8xTAxauBYI01ojsVLCpCUGFyKlWS5iFwiLFtr1KPYqeCoS6CoWjiOiFgGe+w3lPPfp2h4rc6auIx2DZ1ByYhWLeHnpkV7sG+q5ckBIRAYmiKyXSGdV4zbXITMWDm+3zu4oj5VPRCfjMlout8OZ4TQm5PUdKZZhSE1jULG9tPDFL7nWwM3z0HET+d9kQYcEO6QB+ma9gZHp1qZJbXvCIC5fQ9y9El++A6N3dBr+ED10Ye88PLoWPeumyAmc24BDdI0skGk89Q0IzoLqagK6HGhPAso8si9wr84CEi4ncpizVLDU2cFKCpef4Sa03HoMbwfTgERtyptKFRbMT+SpuZsnrhF635h9uqFnszowUXV guits@Elisheba", "key_options": null, "keyfile": "/home/gabrioux/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": " 2021-02-12T17:50:36.668 INFO:teuthology.task.ansible.out:gabrioux", "validate_certs": true} 2021-02-12T17:50:36.873 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpDFqu0Zg94mCjrA0FTB0vMiSBml1pqYCDKHhKBrfaza8u3bhtsmls9i+6U/vIwCEVk4GGPNhWduTJK0h9R5MS2eCf3GJcE0XHqRp/8msUCDPOVSZwC7hoYHZKunUIUaIUajOnR+L2slA/a9JaqataFH7zdrKbPdSKBzf5CXYrc0fK+Fz5OOsaI9KITbM0nESVpNCDzRqnA/aarF9MlN79vpTtvwQ+i0z2Km3cxqwV70GPMuFVo4CGf8pCGpbMrIg9OtoSC1svQ2vZzLkH2WrbDqTfYlOEYJJXaiBKcGhGnLZXfLIGSgvHjpv6xHo63yaMihocyg2MwTS8TZ/VtAaH seb@redhat.com", "key_options": null, "keyfile": "/home/leseb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "leseb", "validate_certs": true} 2021-02-12T17:50:37.080 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzjJ6nlPn/OXP3knQ6orVId2QLswql9gRetKYlicKDBrdFg0MES03EIuAonu0aUR4NEReR7P1iB8eNnezrHyAu5mxIJlZp0rz7ljcBcTLL+11IJU9I9OjBFwzXGqJ1KK4B4Qfw35oc2O6F5V0JP2nHhHgyny37udJLImhxHy3RNmDwc86JhEBCVR3ouYiZuxgxxbA9BXc6+Lkdu9hEMMCDlaEXwNmMyASVU9Wf3dKnMTSCdar2PANquWbpNQZ5cgoTm39xgust5CEyYJ53aOdrti0DYK7qyzxUXre6ckZGrH4oGv2XrDaxpACIiHXt+slvow51gsZnBRHeayMAxQzGQ== hchen@hchen.csb", "key_options": null, "keyfile": "/home/hchen/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hchen", "validate_certs": true} 2021-02-12T17:50:37.288 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDCqZKc+/7iPLhapA/7SYmLxwyKEM95D6eFfsDBNKbEUey/B1OlqYSo+yKB/LdFDyUcd7shMDnhFFg47lNMANrsZdtE4iMPVAa1YhcrTgNbV3HSp9cthKixvU/gkt9Fnb6PEckRzr19wEwxtoNg3Fh/R6YCiSjPwqHmArtC1G2hzk560EJpozx6xylMR5IafatNDoZVw2YQ35fZvHbhHD6meQELducHJ1bKEEHxVIRSTcmXDpBz/vAvBoQOEPuH6PmC/oSwL9HVVIsazf65F4QfK98d+8NCK9TemgXyGDk3Sjz6xdm76yBhFzDQEAzO12U1YtS7Q7zGVsjmSDav2dd7 jcollin@earth\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC89dHSsgjgtIjJ3e/pcMthliQlX2vtdXL7pk62nD64230Fm1PfOVCnzn4LqDhBbGgJT+BrlwOPfzStpLgnUT3l67S834SJzhvSTTpQp2pznnQg1rZOiXjvNhbQwVYV4rphXWqay8weWvFIb5QfXE66mYA1etZtcOb1SMY2ofAjUiA8bAQS5aHpzcMVs4N1A9x5GJx/7/Bj1h8zPpSXikDYasJc3++UMGc45C/AAf725YwijK7xMnZNXv8uTBoGzO/Ko24R3VdvtfMVn4Hi4/FwuRd5bRYcJXR/nUdtA3IinqvlnTpFV2LbfgVTHiUMGhvewRTB+8S8ij4nnAlko3H3bhqZZTk/y7d4K/IerY0mozVIc2x8rMLfqhF9OKv1QFB9fskRZcF/ENHITnmLqCJPg9rgmyv1gCQ5Tbds1OI+IYQIowdiwtaViiTSIy7/CedbHqMz6aDnPsJFMs24lcUXe2d8uZ26RqMinMFGXg0zF7tifuVmxj7zG8DT7MFORDm5f/Hhkng05R5XZesStTBxpEU0MGuzWhENt3RuJdstqmY+wkDVW3XwjUvKG/gVYuJYfnBAsRZIZezw2wfvodhc8EmeHELvY1YtB3DGTBXKE1lsKGN7yV5+nl4UtR2IGzBn9GovWVi5ooB8UlSAeUmZix/eAA5teL8ldFrDht7+AQ== jcollin@teuthology\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeApVHu5er+R9h52RLSVYVhXFQSn/nMN38uWoDYPw2bqYBLNEAAMcPl+rnqHubELpXTGXjJYL4n2mjic+2rOLZpDgPLmOOm4PwgYstJjE1KLegKqVzXIa9HfASKMXaHSgwtC6P5vDQkm2MNl8vXuTAITF7VYdDS33Kk 2021-02-12T17:50:37.289 INFO:teuthology.task.ansible.out:ke6ViJRZI7szYjbP7z9A1fy26THrvnuBT1Q18yy9vgk/sKo8vjLQfHSLvO90CJzouNEKqmkVpmJEqCPbE+s6Qr4+d2KY7L6Q6GAXxrqXK1jSDrrPsJguE6zZ1oWy/7XFMOr9ssduFWocnjQjPtQj3M+V+7FYAsf5uqQnzeMQTjO6AXYxTI7 jcollin@stratocaster\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDk/DYRp5E8Dla/4XHCF504X20W0u7CijZSCFWT1/5082NgnYeR90Prth3p+4xsw3YC9H5pEAS3XwcbdUb/NtWvZVIAIcMd7NOOEClSIj2Agk7NJxZ/4SfQRLrK3H2Qli/MVnrTxnEa9oeS4zqzv9dKRwFJDUFCoH2CpoumFVAp02uyzk7D/+XqFrXzMJUiSHXfl3YM2rt/vNmCtHtVp04UcG0Kyl2TXY37frRlSOkMMu0zbIC551gOyae6NGYhkYHa7u2SnngXcKM78b8MDvMv3QtuUN5P4WpH5qFRdlKNKsQGrzcQpBDY8AKQ+rX0K1ENhBwer49bihYNMKvjYmvTdLTNWgPCu7dt8JSgJQ8JMMhve0pjvFdpcKDLp0fvXDvJvZdLJM1ID/66kywJspHjShuqQNXRMfd1R9C4F20c+R9LY6eTGwrpLG1DNEez1VBHmOu5hQl03T2PfaThfLfMHRTw8JpGXGLMQarCCTm1PC6Z2Iw7KzorxP1eofGA5q8XoCbwGD01uQRZrwxPq0SHzTAkW2C2M8oEz+k4Mrx7+vGYzSS14hqxrzk91wuYhZCAKuGpqwm1ouJ4wxvWFewjbxLgucMfGSp09zJ2XSYRmtm7SAUifFob6Z9czHGTQerHCOTPuYqzlap2gCCVYtEvQyNB+4IqK/9FtEXTr28x0Q== jcollin@stratocaster", "key_options": null, "keyfile": "/home/jcollin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jcollin", "validate_certs": true} 2021-02-12T17:50:37.495 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDdxi9xdXNBYctMHH1cuuyNcPC4I4xVAeD3HZsKH+QwIILiKOTHu/6HZg7Yo1WsuwYSA0MDc+5LCSGu7RsmOEVpcSQ4gtNoFVwsThDyI6yhsm9WQqqXwSPs7ZE+urqd/BDfQ/7oMqJm5bViQMP0vmBExHuIFagj+8f6nspkE2Z7lBbaw8bEbAJZFx74oReOAELsL+Pbtf9Iaqrap4Z3WysmhfP6Rm/q7mncBBmo/fl/bejhThr8fghTPJb81M+XLhMk/rAdVrGdKV9iSt6lLiWdvf0lpTZO4LdIfJSBXr1Xv+lHRtaGXID2ahH7zl/W6dM4baz+hU7ZrbP+vdKg/SVH xxg@xxg", "key_options": null, "keyfile": "/home/xxg/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xxg", "validate_certs": true} 2021-02-12T17:50:37.692 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCsA63hT3ZAwQ9hsvfY+o2Gy15M1GRnRw1ZBSXEiCs9OJku7T9+LCvvH85Y/7V+mOGcsCZKqVTd5Vy3sHAvAF3M7CYJ+5IIDJKCYslC0wkKcX4jj26aMToAZPms8uvs527BNgKOcvwgIfPjs4WJG06fey5AhsuuL30etbAvbBudxRAKdZSTOsMveneKtXQu3Dx7y0rjfI/RBAhCAvisnMqYFUdajoJIl5gVXl6+8mauUOP5O0mvyvJH2jQveqZXTdsIPyp5mq1nA5psqvivWGz2bEJ+owI0GjfuYbyAi6i+BL4Bsd3GeoAs1G3Dicfr228VIUsT+R3xLBQUj3xIsVAZ pcuzner@redhat.com", "key_options": null, "keyfile": "/home/pcuzner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pcuzner", "validate_certs": true} 2021-02-12T17:50:37.909 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDaRX+aBLjb/h12ctmvoMvZro6NrrYDuxROF6QMVpEVucA7jHhwCC/j9dJAq0hOVBO8R3S0bvKQ6bmnCb/4y2J6tq18xKUFCrChNii5OQ+AgM5v1353oZ9SrHN8p9ajvS8w313o2OpXhpaPv3khVDjKZsDmeH8YL/Nx3WQKqIJC1mI4jrI0sZGGJLifEy3pPkO0W1KmguN9AwAWK6+kckAjMa3ElkfK/xuAqBYgNKQUxP9oMgTQRcgpWF8Nzb0npN3v+HOUIRrth1INSchkDKtWob2T+0Zqxwm+Vpi0iFWwqjcN/OVdaDGf34//A3M/tsiQ1g9SqhdhZJCfu5XLmM63 mkogan@mkP50.redhat.com", "key_options": null, "keyfile": "/home/mkogan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mkogan", "validate_certs": true} 2021-02-12T17:50:38.105 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC56FiAgzz9aF+qAMkw5aGoCyoSA8Lf2LPaz3QxTYEzmQt0ByXMmVw0cgUI4gDt7hMBX6LoxPALZ7C1aH8aPTpb6+hxHGfir2tqQmU49JOQiUvBP/Xtt1ilxMou9Wu/ZadEhIZxaP/TBncXNls9cbFs9bV7Q8J3CLCTGRa8kilEpZ7209c/xQI4yQ6q+Q3H4FkO0YHP+Tkn9mg1df+BKkXfSBycjAyOXPvljoetjw0FHEbJ7lTCcSG+hCb2LEACo2OK9eWC3GYwDPKZF1Ht9HfDvg4bHfY3OVOixYiZSim7dbj3+VHWWsiH64usGH1o6fn4vV16UhVJiiicPMWvllzH vikhyat.umrao@gmail.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC9G/I5iaw5d4ssyJ4iTUxSvIBAX+P03VVHsNMHAB70rVGGwBnBJFqJNbukzxfnsqwzN6vSeuMy7I1zc2VVi4GtBQPK0iCbVQN+Njq0VCIxb4hLmjD3Vn8NgRp80955VJyYGT9+/JbZY1UCdPBuld6hsnTipkdo5U3huAHD4i1FpYKol+4DCZHAO5wNilCbests0+ZPtHYe53vnNDz685cx/f7RTtEarvU2fECxkurlBBNvuIl6SN8bo+NaJ01Li0xAD+2fpGz6Iu0WUEdWTkt78ubvA4If2k8XAKxkOKEvfyX+AtTzdHxWSQu7ixkbuksuK6A0bnankn+DdiP6Y293 vumrao@redhat", "key_options": null, "keyfile": "/home/vumrao/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vumrao", "validate_certs": true} 2021-02-12T17:50:38.312 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDZDKRAePFqjugc2ljw4QmM6r7B2dPI39OBB+2HKONtDZI4q+uwgZ1yqnhrbJaU9ux0W5wNzJyp2VfOPhw/9G9jo+efJPg5IySnTonhi1zeCXNpBMY70dJZVCyGEDdWKehvFM9xFBOxBC/h7gOyKqEc28BKoy4cHfULzm4er8PU30MyFBngu8NaoKqyQ3GYAfWOAXF5lUdalbJwl3a+HBb8PjUhbGuznAZF1Ji4JgnNuEZdlNVOKwhEkAFZ1JwXvgLfrEJVQs5b2V9Sp2Vm5ZngsLg7b3RzV5I4B3Lj7QTTraV1zLJGiS5iRdjY7D4W5Uxi2k1X7vaOjySJWWadHyeicNNBhLSfHl6JJ3aMuA8oAAmtgo/8IkJ96GFtgM9j31Wg8wkeTyQrwSeqCuv1uDFr8rp+OvMClhHnDgUklJLWjjYPCDVYR5KV+iOiHL58WJg/b4h6NsoP91wO8P8EcSaiMdq7P77w5nJxDI0KKb//dC8aHtWnRRRlrIS+PNPqKJSNTaSwOTieXvzplhzKFzu2dr79qs+/67iaNE6HBvIX5jVHWCFV3wllyjsC8WYPh4kEVtg2SY00SDOOrMrqANV8DDtHFwwxQTEjz1z5ESuwy9UcUSQfP1/zsAp2OQzPutKs6UipHyBK3pLm6MjaeeaEWu8fmetCqALxYV3h17O+Yw== miguel@hermes.olymp", "key_options": null, "keyfile": "/home/henrix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "henrix", "validate_certs": true} 2021-02-12T17:50:38.529 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNwmgnIsDDJmsmYzkTAEo3iLj7Jd5Ni2R5vD1Z0NRZ39zJMtbicZ0ec3nAMzFyHrmPnKNFltS1mhehbLbtKNKq1fSo2SMHs11z+N8Nl94F4AY8IhXZ8U9GAorEJFVIxHUNEDqwl55Ov6MfP60UI7rNgy2rUt8aOBmAJVhLEFmJJawJgyPhTrUZAT20OoILphrt81o5bTNfmx++Zs5Dc9VtgIh53hexwKgTvG6fxfKdra9m+J2GXpniOsZjPJq/A0je+t7H3UlxCSu0dRy2Un6l5ZAcoeg74SP50dFgGaxkTQqI6wle6NjtPIdssgrdYTgTcOg9M4jWqAr/5drTUSGh root@centos7-bspark", "key_options": null, "keyfile": "/home/pbs1108/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pbs1108", "validate_certs": true} 2021-02-12T17:50:38.736 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDnvVnbK2SNfL8hTmrA09L41RudKW/F7Ty0vvlHjrFjDkdWXVgrafFWGKQZ8dSR6FJ9JLU0X59xiEo4z31ini/Dt2XozFMIr7RdeB3xpiifuUUG5Lgsasd2ezniUoOEukL1a9KkCIxNgmKJuvtlHCt+ZTtyL513yjRICByZbnmBSSpUiett50X10vtvaNBu4LikUJKLc8gP0PHLbaGsU3uOUd655lkFWjWhgExlU8bHWiAZq18Ss63jDlSKFVSk903GKSNcHXMHNPYaXlrYw8NFzejoy/XjxEfmF/Lv3MeAUhHQWx25sh5aDv48mzo8FaIJsL1MrD0QxF1gllobWAfp clacroix@redhat.com", "key_options": null, "keyfile": "/home/clacroix/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "clacroix", "validate_certs": true} 2021-02-12T17:50:38.933 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABJQAAAQEAyiGmj+nOAD+il+FvXvZSpUOJQp6hKh6UPaAIQvQZkdeNRWN7wcDXMUlLj+TBFWa1OO+9xqFlnbgVMDR4qy5orUBYlMapUb1rutjbk64XeY58b6uguHwSoMd81QpCCIN4KOk3cldV8UiyX382baoZ9Je2hsOnnBGMsRtHuVHZ90Bj1rIITkwajq//KRs7kjb/qhjk0xkipDIcaSwKjzPlmmnYn69noODwDexpvLYlCF+LowSZEdrOUvVIigSFG7cLTDaU2tLhxSgbvWu8f0pwvjsyW7gOgM+VeRZ6uHC79F1qvgdgQSxflddIgrQpH0rzf6RNxQjR3SBkZ4V1BESa0w== epuertat@epuertat", "key_options": null, "keyfile": "/home/epuertat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "epuertat", "validate_certs": true} 2021-02-12T17:50:39.150 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDRZyWiDO5JqWHJdUUBr8YReiKXxnL87juFAfkcmw2E9KmANVH2sn3K+OgUCQfBs1Wdg6AbcL2GyMm9alodUruzOm127/bH9OcGmu+8SNsMAYrAF9EzsKJa8Ba/I05rl5a87p1jbYG9Rpo7u3OvLx0LjgqydRYcmjwo/niifYNDTaN0K3v8s1mN4TorUZW/i3XBMBIBjlIiNGs/I7IZreR+1N4V8Nwrp2EWxDv7AwG3nvokL2RTHcsV0KgmAnciv8nXXjIf+uqOPuGA+F2qW/wz92bG8R0O4CjH+QZt4k6UoUIXFRW+LdOVAqmUrb4W0OJ8vsVeukW46SvPnYUTkZnF sonea@think", "key_options": null, "keyfile": "/home/tdehler/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tdehler", "validate_certs": true} 2021-02-12T17:50:39.356 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEBD7CxJJfbLW4jtuLnTwkxVbhCfh/JL3Qj8oeLEkny1tE7QCeXn1bP5bScWSgLH34Wh4TAuH+8/g/koFwcCy1YuoGBdBV4KV7O9MTnFngdqNY6LKzCipm27tnWZEa10hrF8dRFTmhracd6kmmVGHgvGQPVAVONmix+2IZ83wdj8XJSQHeE3PHFLO39tU+ez/T76QaiBLDVo4ctHVt+N9bBbHV6Cr8fyiUUU6YEKMlcedvxGpnMK8oeOSNuDeh+DldcM9KYzNGEOM34jGRIvJp57dxQDQCtPPW5hvBXg7yMyrv6TcJzDd2LFD6AC/Qw9yrEQPXXyBCDkAlCqDR0+rH", "key_options": null, "keyfile": "/home/laura/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "laura", "validate_certs": true} 2021-02-12T17:50:39.564 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2k8xsrsLxEqJx3rV6ZZwLH0OjYCDB2uIYzcAyrKEOKlVWO1ICKxkw+gZ9tSiPFNLfq13io9BIrAKJwlrdLCJLmlgCQwpwUQqzmRlRZEm+WmD0dl8LPeQSidyJkPkJgYBEshFPjDHIj10Ei1pJS5svI2l7G1wMWxlt2kHL8Og/4zdFTRctxwqZcvzLmRQcs8lSSpqTkhfeKfmrpR+je8XFbCJJ9m/nJbgyCiZKaGN8nf36pCQDEva4rvaSrK0a6YxL0RWN8W3lqtBPKIehxffr3lYDEkD9EUW5vf6AUKC0ioh4jkPo9WCjaoFsOauMOMVQPHehEAMufEO4Kym6nfrf yaarit@localhost.localdomain", "key_options": null, "keyfile": "/home/yaarit/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yaarit", "validate_certs": true} 2021-02-12T17:50:39.772 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDUznqo3VDmcKRL64P7sbjuxXCW/0ljFTc5zZoZ9UvtZvU2jrRQ2X9GyxcrcQnAWlF5YZHqeTDU5o8ixn5fq1MSmaPRBNDKKVQOfR/vFQV73LiKx4BFsC1/EdaPie/XqltaLDTtfqhxADJlz1H3a3ZlkeVHMROTqo/TidL1YUYlRH4qHxWWBuFuVS+EYfTegSZdjwp+DuJA0iN6X99+DfH2DhiHENjEmlbKN13+H+EPzm0K3ya2w8D6ANodB3AxMn2tGs8mLtG/P4kCjoDnoUVFzKTyUpTtgOjis47uSY4dL8mTbIK9kZAiFxnT5sMClu6KRStDjRIrxbTBQJ8wvlat admin1@rajula", "key_options": null, "keyfile": "/home/rpavani1998/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rpavani1998", "validate_certs": true} 2021-02-12T17:50:39.980 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC4NYxUHx8HMgbVIHadruN1kAJS5be0aZA9rvGtYDfwD5siKNfXFRyLG3thkjxtEi7DPYMxgbpFOXW9EQKh5sQ0ohsADZOi8FSIAfQlOQ+7HAEp7DjBtjxoqB31jJ1e293YtO+nqAhcOMyGfDx+71w5sS2EFqy7PscA9gD9aV2iMxIR/JD1LxXabiqj12jU5AOxr7ZQ1+gM9aroQuDpNxY4U8jBTxx5OtIYRkeI/aN2gQj1mXGMKGo0ItFn4lsXGLxGxMpa9K2UFw2dWQVTw9VNm79nixrCX+lC2nyaJPsRYZipXF1ID2NRk7oJ79dJ/uRBBXpunihlEC3aF+nOgEPH ridave@redhat.com", "key_options": null, "keyfile": "/home/rishabh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rishabh", "validate_certs": true} 2021-02-12T17:50:40.187 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCeAhoTum3/VmRbxSyw5gwgcPF1EWFK5mEPX3PZyo3xPsUAJazALfge3aEuCrZBbsjOCoj3GfFhr/ziEF+OWvhBoPK5hQzP0V8tE9cWJwvHmD72vGE36o1hFO0ONE4DEuu5KRDqQJUgeyQWXsP26uHTwIcvMpy8egZOfyshqF+lQVXzsAex7Ninmidsf/sRNLhzT1kLrKT+ZMSbZ7Rqh46TMlmLD1LzJZNcb6PGVwlfQZmV6T7m0COu/skeSxMcy8hqMvHFNvvfFSqkfMw0KBGLDxMQr4L+e7sXwQDnEZHdzoh0iS3xDp/Onr1FSmPyEjz51e1sMHBsqklHWidSZVLt/+giB4o6Wdgzc0um/QxW2Ne2qjtDlViOgRE9Fzcw/cRu/Xsn8qtXTxYyxXswjkA8qX7WR22OYgjYChw9siUsKkuD2yAbPMVl+cZWDeBhQnPMbDMVaghS4SAVt5GK16jij4+r3IE9vGHBmGwJYJ/OrtcxBrUrDopwsGLcU+WHdv0FXuiw+YbfnzpthC8rVj4oeJ2IUOQ5JB0ub7NmbR4QWjc33JfD+AeCB75x6io07V2riRjMD9gLC5nwytJNG7Yytf6xI2i/CprW7HhsDs2ykOVcDSBMnL1oGW6FeHaDRW39r6BXSt1jjZQ2E6pDEBTDO41tPEL9EVvv9TK87GTylQ==", "key_options": null, "keyfile": "/home/skrah/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "skrah", "validate_certs": true} 2021-02-12T17:50:40.394 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDklgWHrL9TMaMxIyN9MxVR6nR3xh9zfj0iR63/yOzaslX0WoVBk7CYTsRlH2GFVctvixVwrWbQgMaKe49utal/3FA6sMl8S2DTrtgC1a/exUTvU0U/+ZhyUlkyjQfRgtYOUwodNupL5NsC8VZqZshgmgXC34uye3bplx3VwvpUt9r6QxJN3/9Dbi6HxudhDCLfmvROhwQYHaFF+GglcbyjAt3uvsuwXVKBd1/vMQwbcUo9/zQlrnmWBRuqTNkXGrZauWigoi/hyuWYQ3MUbRc3LVpJncMv7BEtcj8fCVvPB09SFee3nSeTtFkm01GvRcYd3HBz/YL4Rz92wXl9VIxqs8aaJBzy+MD0NO496Rfqtz6YuZ1m7AxDIccDW+OpcPgYwaUKFXNTuVLWZXhkS0p4swefcKWRlQFF3BLOebN/Kr9jq1MFa+5pX7Gj4WF5uCIX0/fx40KVZNhfl3qywBy/d5MKwXO7PCHMiGcu2FIXdZ334MxdgiIVoX54PpRP4hs= smanjara@localhost.localdomain", "key_options": null, "keyfile": "/home/smanjara/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smanjara", "validate_certs": true} 2021-02-12T17:50:40.612 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDpp2NWr5T6s+jzHiz/FfCKzN7ZBnUSzyE7ieuntzjeGotyupAAjVBoSpCJgNDybm+WkE4dShv5mttPJ2/q1UZCLRj3nN+y3NG333aLZuyugR0ht5qtvGVegg58PwQ8CMobR7ca7bl5UdEy0UIrM3d/qpDOuPkuE4MIJcrtZGICE8NAVnmHNOB9Sm4ovAYkewTuMAv40j2Dl4F9dD6bg+OJfBOJ9+Zj+MpowYlAILtEB9Dc20zB8GGUEtr9yfOVroblTYhvTxGYRaE7aVBe83DNwkWGbDc/qUNR6bpA04w/P+LclYVsVKRoHOs1EXHSwNMFromA8RsKS+rajwu5uRg5 ben@dhcp-185-182.bos.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC93wGgyyQmTxa7f+BZcYPUapTH2N1mnV1ikl9nzJzHhgrtJ6pv00LhtH1HqGhvoqnqZjBD5XRAKyjMRcMBT5MYcXXep5RqZ9sJ6Xz8rHkBjfl/OOQUeZ/ZyG/Ksr1jpGraN2xtrjQQzbqN1YvirCqfLk2exPvVeEk4CGnNybwKwCWob1uKV46ASrYeQK8pfHxrW7pTo+xdErHOvb7uumZm89/YisUJGicO1qSHBeuw1MxjhnD+fIcxHKWY5NNJ4qScW6a5AABk1JJsoHkh9IGCVaf/zJv8D3gil5SiM8LIBKaNa1fn6oObPl/Ky/wkiIo2gmUZawvAVA8iJnX9eUPl ben@bene-laptop", "key_options": null, "keyfile": "/home/bengland/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "bengland", "validate_certs": true} 2021-02-12T17:50:40.809 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC3YhI4am/T6s4rJRvbC/jAvrNcDhE/zaJiVDAjEuA1W/5Xt9ObX52eAnp+tOOtelXrX8ylYF/z5wGNQLjYrWepXhuqaEGxeVayCHzts6LMpmvyZ6sYnAw9HFO9rzrTawm70DuvWqY2Zt4KA6fM+lfXk3Qdpi5RCPDKk1jpXwSCyUtUKSWTDRtYtvvJTcO3H/xCJxQq/F9sU8yqgQcqFPfMNrV4a8gNj9KACqjt4IEIfuxKlKhlMA4yu1VTmGcNuhQS+geJzsEGuxZUQb0Io+q4o0b+40BQtazKgOdTcQbgNtqh3wZpjJsvB4AM+GNPjZY3mCsd+WPGg6dqfU08Voo2b7BfSwyCn14fQHnj1nSwRIea2juFY6AMfVJjstM5iIz7lOH2c3ydWWhzvSmMeoQ47zBXlg8JytWmDrXiyd21BSIpTL0Tn8P95wHYka8U9KRV0ZY7Jr2w/ThPdy2C+Cdir+P+OEL1sKHSnmEyQEeONa9P5yH1Rd1kM/FNESAIRiMLGspzXQeWoJhHswVaKoSBW/7f4b9VENxda8qAQi6ob8A71cAjbpRrkUmd5EBCSFMNs7lscus0/deUVvlpsrMjDAJa8S9gqhGHNU0YvF1WBIz3paLRAHDm9DWrHc5lreizJzX5ul4P/Z4ITjpO+9ZxKOcyi8wnJ8Og+QUYLpqOIw==", "key_options": null, "keyfile": "/home/pnawracay/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pnawracay", "validate_certs": true} 2021-02-12T17:50:41.047 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDQu/CuhCIFVGCLbA0cjn+qqcwX12cRcFA1sHyJtkj6Ltfmge8FnfgAsex/LaPJwbtbBiVl9xRcazHNfBaDw5DxMRSJF+e7WTydZC2Jm4qIj4D3Lr+imGPrJTK+kjN0QXRXknGsNn36oRF/I41cGqYRGwawFGSPZBHuav+ehDk1doTrbDlll4zMUWm2iXTcBIcJo4NR1LoE2c9/MPfbH7jKm5ke7aS0ets+EANEplkeClfCVAashtIF+Bz6a9qk4F2n0baLT/s3IQkXCwwO2fxxY03x8gWiB1n4E7sousEB+J1Skl/z0bwmaiY0C+raPoV/FyZxQuh0ocvdU7fnQbKuNoGE/Usfm9Y3g8eH81b/k4fMJL2lqXHkPft6zUV9bJsp3v1iGEkIf2+ER/QLczteNz6kw/uTPQctMEDXM17GEnBXISVrQcPSW3i93eFwWacs8iT3G4bjEQvpxSBQrhDdCCoxy8bVWqkVeDWw5u2q/3WL9Ov1NrY+G4/DYb2qONNQds0vTHCGvemU5GNjtaeCbojkZ7Z3KOeAY8080kKrsO7fve+ZMACCIFD+se8kF7NN4ZJASSqq9lJpBWvtbc0O8jCcVGNuN7exXjrcbiN1L/9HkARV+KEJU3jbXObAhAnfAaeRWBwmTfZodCu+fDd/Jor6l36tjD4kC4/CPrDtyQ== alfonsomthd", "key_options": null, "keyfile": "/home/alfonsomthd/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "alfonsomthd", "validate_certs": true} 2021-02-12T17:50:41.254 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCoL9+aMxLtUG3aRIuT4kbjc1CPuQ+wbS30ocuJ2YIExlKMSxVr+LYqbjcz0T4f8gAKG539dZQgD9BAsHM8yfOVEvei0KsQHn87pzgYGR3DtaMx5IEG5xNyARPu+VuEoe8qlL3cAPwTiaU78datD5Cz9GEClhL1Fhn/Z8o7BMbs5xlcUIQ/J+s9/sKMcqmmBJy5IHQIEUHCl4SFwsev5g5ADlnLNNxjpalExuBCEeoSma9F3b2YEJHvpA5JhCIMZpVfXbgh2UhUsyfAQN6KgOIQ/XlKBMQDtizf2vZ4b8hW4ELcvKSADfTsyyqEs/Ql8jEJu32GFKf2mYPbgTS6E7FZfQz77X+5OBhojFAstB8LSSrH80FrG1gCbVZLnW0jXcSehKnU0oNmUWOKxxaT1b00fo/iZ4P8O1uJRY17/uaVCPNKdGTZzIlzyC16wsFFBxBl0d7HfojQY6qfww7AQJhvDkV2sP0/MIJzwriWOzZEmfr8Osqery5vFiYcSvMbdMZDfMEYd3z7Va2RiTIeyXuEYp5hzf8eyJRP/f2FEw8/BBrm5JCxeSWUx7qwgPHrJoWAVxCIg/M3YS/vfVSFc8Hh88sgc2upq0OqYr0uq6EFzUZi4aoOkOVYn9u/uc/p1AR0KbiQOkERMUDWTyADxrHaL0nQmGy+XyiJQjhq5ckZ+w== doliveira@suse.com", "key_options": null, "keyfile": "/home/oliveiradan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "oliveiradan", "validate_certs": true} 2021-02-12T17:50:41.461 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDh9bzZJulXGES+l9Xh6Aq15RZ8uQCGuDNhlNQtDblE1ISKJ/DqGYXp6wUW54+oVNA7eZiXz+fi3mq5pPEtZZOfd3ixEDzDJ4E3cVXoDZCqeWEmea6KvybeY10YxvB56TEI8U2KKAd56PRl20klXpLCjzXNqG7n0aXFcpCMbXDu369VX6lOk24K/7++7Wc5SttcvVu19sT2kqzsB/S1Y5cxiE6RM6wVtqBoksp/kRCIA16ruNwx3GUabDfbEoUGXlkkUP7+TZgAbHtBsYy6mCQhwi0S2+WG+HhHDUPjHhV+MdN9ffibCtOEGo52itVLVky09VeBocuA6H22JDGXPBgjcgf2NsZIqcKqGHhUkXmH92fhRSFOBLKHstrBq8jWRP/mNrgj8cQksDsakQYQbDg5dyabp+M0/iL2Q3YVq7erZI8aZMA7ZF3WgoQNYZg5E7oejM8URIlFP3x1ne2ClRC9a74phSCxeU/NVamGN3G3dImzEXGOSNyRggHJ4jGrIGc7tLPCzmI5OkomcB5OxReqf0r1TNXuUAqw8M4EoWt+0xoAmH5zVlUHf+psUCJIEV/4pbgtoJiSNq+LVY4jyDEFbvTAL7MqyXorMV7Tqlj+/3d7RXfhW9lR/SHb1Z3jcHvz4ZzYMzKWeJEjz+Y0NwIdFDhcPmUOYtEDmuRRrgYiSQ== sebstian.wagner@it-novum.com", "key_options": null, "keyfile": "/home/swagner/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "swagner", "validate_certs": true} 2021-02-12T17:50:41.688 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDzYzqIDimPjMOUOUz1A122eV52fl3mgGgNBRYRlOWl/lOxM5oOat5FPkJlnXTVyQFl0MlnYgIv9fTnjHVTM0wFmFUItlUl151ZmK5XExiGpGVsqG3UWNyKJxZpk9Uh2a0T/5CKzzwXnv2fJMNxzfqgopYDrgq4TFEZcdvgWJ39dcRTXST/rGJQ8wOq9z7UNzKU5VOGxvgi6f/nGevGIHrSaTcY3Uu8BW8kXUrf9vfl+ZSxVjuNHXF/VmrJH5Ne8mVu7DzPMrl8ZZjxsfJKT/eQqF9kjqht5586OSPKIwg/LEOyYj7Wyyf3QbDAIp5bCygQfmBBl+kIxphJYq3EwUZT ylifshit@dhcp-4-102.tlv.redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFgwQYD9amVSYq6JzjHyGmKBJATQCM9xsf6QnF3yb0Lvry2nzY9F61evVXQ95TmF/AqasRx5p2WansIvegG5ag5mytvRwFXtHzPs4o/J+XjA9o4+A5l6j9cOFLIDD/TwrwlUjgMa2PhqUztSpI1op906391RLrJAyflfDghJxeVl6dJab6TEsslV0W+yaCS3BBxMXXrCbxCB5Ms3KCRKDz0DqbhFc+Nw05L1O1pC98HBFRIp1SaXlMcT3++CRQ3wugpOMkDMNLbycl+vdMBFY3UZ28rRowS3Ta7V5EARZVVH7KxU5cL+8Hr/8DE1/ITOmrluaEZ1KwYgoxZbZ3w7Ptw0FK6aiCZ49YtqKT1WakoVfO8m7pCdxkqUOA91loMQtxvCwZ/Vfoxaut16gklmPz8N11xvo1H5XB8/ZrvK5yKdl3JDl+wwtfE/fwo4E/fte+rPRisoWBRsmsx8EGxs75sXnb6n3VjZ65vYEKsNMLg+5EAaevWLLeqF0a42UWGJrHdb4VKX/OAplHrOk7kA/VwArow8f9n+iaRp2hgDU9OhQiFVRhgqwahlmCFFXkcP4/4WlxMCuZrsaNMZgMjVhfsp7cskKOezSfZZey2MB2fREMS1wg0j40CdYRn4jTNEBr/LVszsoM7t7AwXUpnqXeYIe+gePx1PJb5XSzSPZ9Pw== yuvalif@yahoo.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDDp0xz0ml2XuoQe70TxVeTo3Ib5xMinDrYNJ5eq5XYNa8ZLGcEByeDK4myzixTloAo31Rb/yDyGqv6kQ6nFChihO2aab3Ni/bsMOaJat1scrznaIYXGvLvw5KYx/qBDUxb3qW0vO0N/bB3dUPS1 2021-02-12T17:50:41.690 INFO:teuthology.task.ansible.out:lXTJaVf+KuljUk7v38ov2gsPWlofRYZNkPGQfUKXFEXc39OVj+Ex5LkehH/Z1LwjRYJa7jAdSm8hE5m8m6WborrzAA4/iFuLbhtM/Bqm4boSqa7koBVux422QX/HblSQ2KAeT1GFOS7xdcO6ndyp6mIpCZWunTjZwSLyK20NvIV/AzdAro4GZWi0D34PPtbSG344ZT1dSLeDRhYVkDjS6uutrmw9C+Ns1bo8s1ydhxL2XEsU14SWRWlbyrMF9PXyJOsaNiBNWGwc9lTly0ZlmPnN3l0AC3RIKOstkEYmI+TMj21r1tltz48e+6dH1an51wwnhNRYiODd3hbS67tmIg05BuSU5+whJ93wzXw2rC4FLE= ylifshit@localhost.localdomain", "key_options": null, "keyfile": "/home/yuvalif/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "yuvalif", "validate_certs": true} 2021-02-12T17:50:41.895 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDXa9uTf3q+7tGnHzSxv0WbgpRKrBIYjAZ5x3V74vQUSD1aAbmyvOqgPnts7XttLYImjjRqiisyC03L3KgR4b3SOeyL3KFxrk3KkHAJ/AJH+Y/UafyDhHNl3gRNg0B9rjsa7h2dSlcrFKnAX0pDY1HapuNPrzYWrXsEbo7C6wteFh7PyPmTkHu0NzIIPC4FcLl8SGaww/HeBh8v91IymgFSgXG0jNC063FEExWdti54yNLGh3gpW0FytsiRN9eEOKUOgcveT1tejz9ORv8evO4Y9SCQ/lsyOKQKNcG5C7J8nsbwSjqQ2cf6i2aYFZ0Oq1UXuhJ8APJCoiKgLJy7AiYf kkeithle@dhcp-100-19-89.bos.redhat.com", "key_options": null, "keyfile": "/home/kkeithle/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kkeithle", "validate_certs": true} 2021-02-12T17:50:42.102 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBYa5uEOlFv1s9K9D1SbcaApLwX3LBwASSnRXDHI7MtU paul.emmerich@croit.io", "key_options": null, "keyfile": "/home/emmericp/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "emmericp", "validate_certs": true} 2021-02-12T17:50:42.319 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCj28V9sMybkAzhkiaNiDDr8TfNVyr5DNdbOqszdGEVLy1Rxh/UYnxUclsR9n8lt9VnCft3M4NDTnVwlxOHE8+0oolJE9HP+DQomsgj3abP04EGJyOA0X8E7eIFQNLRbMFCD0yX/SNNlYPsS5fPTFhIa/gbBPmDBE07kULts66Tvc0Yo/2yIfwV1DV34bcWWVGi01XybIHBl9qbvlfBKAUrFEcv3ZdpHNhuZw6IkZY8xoYxo8iF+toLYeGX+gFxqCZUUOO+NBVmb0hVL1LI6bU66IrMOQDkFurqpvoraBTJqwZgXvMPceFUkcVIH4Nm7cK1X2WyMtnSw2XBpLMT8C4V mchangir@indraprastha", "key_options": null, "keyfile": "/home/mchangir/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mchangir", "validate_certs": true} 2021-02-12T17:50:42.527 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC1OVCo0wMITAp8PNUqsTRPFz0SuBL0r1RzKFJUoxQm7gsP9qttZUQRlfiuZ7YRzkuGLOEBngPuIcV995o0lZ0+LVHA8jxWEJamb6sM9YgnonbGKDHfl4lT8LJd4YtjcWnMGvUUdPiPgIkfrpxXtx+RzJ7eCTmgRjAJziZrRh3AqXuKty+s9MDnmK8EQ2smpLdkMtZG5W3U2PiVaGgq2FIIOAoqr58xGqFgZ4tIpvWFpNxVsDZXIg2NdC+ouuKQvuJRWq9eERRHwq26CR+kDFNZeYSyhYKxNBqauLCsAoXNFCJzrdX5Oi9Ox4lJTsEtYX+SMziVipQS54F7MnRAUccukQ86IbXz8C5Ok2EzNo6smwRZ6UPKRvebWjIvfbeR/iwPRnrWcNgjGcaTR6yhlRJh8pBkW84nyY/bJa4WXsoNLgutdsYkjl18KKhbBldjdf0ovfL7FVgqYfAo/wpQYJJeMebP5QAczxvZIegjHRrAvbs/nbQTmuAkes3WL1hhBjSTwyIbWCzAuBJk8MNTmeZkVBvub7XoIyIySppfoJC/1a+ZsxGkzAlgrIxXPYrrBzTtXd/9YRiIgCYx3UtfdbPkE6hT6GblH6ST6QESfWyyvfGxwlOmu+F+icRyVVJbgM9T/PD2W6I04u2/1mcfoHwaQGdwvbbBkbjWzm+zwNSisQ== sanupkri@redhat.com", "key_options": null, "keyfile": "/home/sidharthanup/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sidharthanup", "validate_certs": true} 2021-02-12T17:50:43.155 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCkeYXWjU7wVdODC32U40QRFgFH7ZEuEsOeWWaKx3Y5EBuAcwkTHiF9Bq/2mrDu+c9W46S8L38Ogc9GZ0DcCXtN/Z4z8iKOIIi5U1GzvwdbKuiAPCd8bdYP0vo0MOIRgylJo5hcnB5BCwCXRN1hmci1ES5UbSeQg0rydMiyYXHZ64AT47+qVUpn/mVP6PVw00pPqt16l9YxhbPFG74Y/WnY4FyrqXGQJfv0tcP7+UR7GQdFiI4QQV3ZUSRHRLWGcmoiiOmGimLhtzEmP4Jzy4gF4wg9q8jyX54NjfxDRZs+YuaoqqWKz/HSz5ic7W2RGKHrRby2N9wNu19nmC1NzIn3 varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCW58OUR3pniaYgvr9xAExeNm4eMxgaJfFbEfEYPztTWv/xxqqihBYCeFltfgWbu7ERTy+hVNo1PJWyyczjl5Ivc3FdRmhykosygs7X74PPh02c7Em4MsSTr7SSPNN20rkJbWqiYMMOBbqbiZ7JXiSUNVilLHgZqZBEQc5Rf5I1PyBLHHJ2ZruPKnDPHCRLdwJ+MdXy2haKmGrQLhtzIKaDzh8VkfojL454k0aHCIh34Z0oBdP9757mNuYJUVEkKZF5t2oML84lWGMgwZxUWN2yO9ALTti4tNH1fXA6NipS9ztE+3hEJe1fWXEoUQDX9JNu4qEHTeCz0secQMkyZVDZSQk8yDUQxQ76yiett37sQB1WQR/3fclcahmqQ4dUS7M3Onyz6k6zia896PBK6G48TBUbFa1rw5Hx7WfNZluDMkv4YxkQvfz7XVzZx2/lhs45Kjh1UR9n7LaW/NID98vokcvASw1N6Bk/6pOd/45iPyEJemwovnQLs6ecwlMIikM= varsha@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDcQ8xMjwVf+KiFIQARcbFT+wg4NHysBRrFxk3Bx5X2k2hGKcEYR2tY/ApHZYfJ6me3LUvJEXg2c/6xa6ibVTUp/90l4y0cF6RmF5uB45J0xuBYoFJd8xJkVrhDIlZix4aFfb7TrmJEqjleBrkTwam7Q9aE3lss16nVOxU4hiViYiCANJ45wobK6tz7y5btfzrHE/iYpwniDvzLwK7EpkEK336VVd6f9wV0vG+ErEFxnNVxheXLLm05LuvdVQvFqJcTFrc7/GZ54yKg5IgwVwXqSAyOXIhgduUqk+oN1ciA7jnSqdiOKQPS/VmYkbhw2YP85l8wGVreIYUmOG19QEyVLB82oB9KS1LezM2KCH61XW22+85dHf4PvuL9DULhGt6wBCvpO10OHxw0JvHXbEgBPhiyd9j2yJmcQozbRPth/sMUvLKNXOGabZhhkPBYL7xZICHc+UwxrXs0h/j9jOjLyHzBAWMomF2lzgbvtzjcbxJ1a5oclJ6IW8yHJllkOh0= varsha@localhost.locald 2021-02-12T17:50:43.156 INFO:teuthology.task.ansible.out:omain", "key_options": null, "keyfile": "/home/varsha/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "varsha", "validate_certs": true} ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6RDgqwA0GrvE6KqRaGCgYnFhnxdi9ehjIWvfaMa1oq0CqYxUAee2bNbp/X2Bd18Vqk8iH9fpE0+qrLgAH/8BpD+MtgGikCFMORvazeUtOCF/OCDO0AMg1jvl8aTdvrQU8ZTDPXa65Avpkc9vwugcoplnwrW7lKx++XqCWsSlfHgEdyMV3wrt/ZDFwI2AS2bZBpWM6ViG1KSjXjgTi/T3hSZPY+buYvWjzoEjQO0946iI3iiKKCGohEpCUnG02oJXcS0vvgl8dWE71451lbainEKJ4hbDhfDmy57s6VXfVUANsczwiPWiSC/ZmMlDaCCBeKHO0Gz0xXaDXd9j4SA7P sam@Pondermatic\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCYylAUS12hF6XndDQ3CkrUddKmXNSd8pOlHwO+pWaDTJPKxwtPL6sM3aOtaAPLrGkv1zD2k3iZaxmDwfPYGaXOyb0KS5ZCddtOyNXj5JLrjxh2niqFPP8SIkdHLOP+0Hv1/tB0Jz75RV+tNIT1SJTm7OZ8wZNy0qYycITSooJ14iEgyZ7rOa3EEzbtIVpHS4+YqIuIJdV2oV5egCrJyRv4sPUrS8ERXD4QZSOlHE+W4Tf+WVrEfsHOdDLRlAXJUzhuqgWtVouiYcW5ex294z28eA+12lKaPQpE0cft/NJ6FLs41yqh4ZN588bl8Ada+yTXEaHWioQE 2021-02-12T17:50:43.158 INFO:teuthology.task.ansible.out:EJMlca2jIgrP sjust@rex004.front.sepia.ceph.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC08RrDQ2M7Zjcpx7RL9e+tjv48kBgDNyTenX1QdxW22JA2JR3D3ThA/ej9HtiPWcIVz16oZ4tzrCEVwaCMfbsmXq2z08ZXYqwiBuoXJVshU7lIKVqAtadT4F6Q9eQvxEo4zWdBib1W0/cqNLlYzU/DuwlUqYnWBDLGxc4fD5Kxrz366oWUDB/ULCpl77TMqfTKlStCgGXot0zLqiscqzq1W7XsuA7A5vQw/JTq7MvIME+SkUuGWX0WKwTMIRZKNWBGmlf51bNru3GmmAnKBUFsJv9H4G0FPqGY/aH02rr5dAgTTmfKt+XbWRfMTMpjJqVRGX03+1mQNoYQ3kNDz0wl sjust@office\nssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA5UMPySf/h9r2/OryRXrO9Qq9k5J9Dn9nvJYyaYCE82H/qkBJGboAJgyIXfbsGYO5uPM25d3fGRH4magzAQm2hGWNbbt1YWn9HaIyKCCXo9JX600/dv3OZbHyGGmgXPa4Anl16j1c2vxRRt2+ArjlyyhP7cAcRkilUGdagK9u6ALrwnz9hhXYV+BmXrxMm7zZCmdWHcWz+YxoA65es4+w+aLvVZWmUCZLF8uZrcpPSw+oK3SfZE8JPRc5vWulXbWEEfIoHb9AszPIpSyzbfPWXelmc3XlG0KD66t7w+AnNlQPt3JVePRCfFHPYeTj9B0k8sxPyUqTQIl5pu6GppRMOQ== sam@Deepthought", "key_options": null, "keyfile": "/home/sjust/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sjust", "validate_certs": true} ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDj6DxTZNodVnwJZrnSZ/jv9Cw4uYLb8MGpccmOvtwl93sVBvqq597AYnBqYiV4VBbKU3GVFbT41zkplia1iCfXoddttCOyWmEKviBXRnAsd5f80dxL0pfzrBJvCaH4NAfYgVw9JFWsvT981b+nB6vum4Vz8Dw0PM5GPnVUjsZNFkpmDPUBL4yHPmMCGP2WtUurJVD3Gm9t72XlG6Dl/vB9Qw/i4IY1SoCCbuhhNgE7IN8JXeGmVJbRwDeUlP+Q8Ti7iedV7YB/W1opt6eT5EPzX8NcxTc0c7rrDgsUWjFQYRSup0GLK9ofdHs0E2RbPPO5OSc5Z2+B1IEV7fZeyJwAyHwUgwx7sSGy7aROdvinaj/hOBglcgISyQ2CY7LTV8NGWJquo7yRCJXhZVIHfgqfBcrgWdNbr+vNBXnWe5PCtLhV8ryHPSJ/7Ru+7voQxgvwux5SdZwxXTbWDKni1f6/8bNrhRx6bmLK9xuYAgt2FhlIivI7L5kQF4etcBiaf68= 2021-02-12T17:50:43.159 INFO:teuthology.task.ansible.out: deepika@x1cabon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdqbYcbJYtgCrAbLo7PcS82caXxcT8cpX0RkULaFXfaiUDs8Foy6Yf/XoEHIdpgw8kX6qBdydYXxti6HFhYDNsL5KleKu/yA+bVKlvri39CRNkNxFHAUL2CRBT5Hefj1AJQBm/BKVEN4/eb05OlJJ808Qb/aNycot/YwgpNvUIaO2yTMA0meq82wNdZmUcmyLB9QXz4QInTcoAsxq70KxX95YC938rTWpbg1hM8gzLkJJt/OD/Z7TMYv/Sc5KGOFqo930AmObQ5KPmffkWGYbLT65TxlCULnvIr63lWHdGTj5mYjib/v9ncu24az9GYRWYcicV0B/def+KVrrIp3Mz ideepika@teuthology", "key_options": null, "keyfile": "/home/ideepika/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ideepika", "validate_certs": true} 2021-02-12T17:50:43.355 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDC0h+TpaUkm2V6Hbf7aPCTys/nsm51CmY9s+bdwEXAiSAWpq/7Xi2LlN0mZgsrAYBHbr5L+Pz8x0YdigvWysUK6U9XRSPM28cbWmh8F25re9HEk8ARESkMZKTv/hKFs1v7kTHbT9hNNynbQGQOqEaazjr0shcli+Byvop27riX3QDDSsd5xfCjjyDz9fiA9YESlF0kbXyFUKPsoaBhRv9FUhtH6DxlygxtxTsKyKgeoiKw2sCfeUMckcJKnGCMtkXW4U8h7NL/fPZFyBaMDRi/TtoBKaVGLaWqF7RuukfjoMCwhuIDmuTG3WhanH4HJrfNenPZkpZUBhzWwZcleao8SD3mXXfZ9gcZb8QB2ymEtTqrAEyDoNWO4q8ljNP3WJXIgF0h0Sjmx/CV/i0ArEYlBMwBD7DFVnPJqCqSyypuQdDGsDNQZxaRCiYg7nIslFOpDr+0PLp78m+CHyYbSFLTvVVJoJ1k0YhMnDPHM29nt7fMGx6Qrigwnuk0yKg1bfgJ/t0TOqEw0bqGKIvHnyoP/n0ObazEsiPl5pRr2ehs/Rqf35Iwhc1i02Hhz/3Sqwc1eAgqm4/OVghxqUJGpRocua5Gf/vO32wwoyGxh2sl7y6K2pNDA3glhKYbZeIh1fjnaTjikgleqIQkCBxbTeAE7n5+T6TcFtnII6RSxYv3tQ== gal.salomon@gmail.com", "key_options": null, "keyfile": "/home/gsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gsalomon", "validate_certs": true} 2021-02-12T17:50:43.562 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCrGYBGPDQqn+2jnxKhDurSrAJvTT8dZMYGVCt6NbjpFhDFYtyvrF8Jv5NKBQynG18LwscN7a57L0Oq0e4jOEPRz//CDcAfP+L6p45L4BdUzTysN4+rQ7waMnIiYp7VjSswhGH/BnYIVKwmDqzM2fo1w/oLIT6mgwUYqAdceeswAkXBt7jgagL8ox3E4oHBqER7f60AHdTviuW953t5luVQiMjyF2fOwZuo2ycufIzBee0CwS+xyRuJQwdgD3HL4Qs2vtBCX+H8n3wC6A9JrMA5tzCDr5VZKBJZURIniwvaXfJodjO4SvBL5jAfXoMqriv8ZWHbap2fk82YE8/uPqNvSISjixJEBgQinklZ+iAOYm5tfPbhPM/8ErpN/yJ0EoGRn+nj6X6nAjXAIk3AENopeqtM7mLZ26RFIuEV5ld7eqUvAfiy6Hk2LH9Q1Z4CXaxD0gDGzUxgdAaaAgLAR4btWi2n7Tb7ZNDdOfctg2ya9ftzZBOHvCaVXSpJCry/GMU= skoduri@localhost.localdomain", "key_options": null, "keyfile": "/home/soumyakoduri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "soumyakoduri", "validate_certs": true} 2021-02-12T17:50:43.770 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDPeuSTD7kv8yQX6dW+7Vpq1mqrcr8JGubPYa1DpGTV/gE50d5L42UspUN2FXmICDMiV1h4Q9WQAFma3168f1I5I94qFkV+/56szCMoILClm7IW/EV4t8fTxoyOpEa9l4qSU3qgJpDbY04CkiyVJFuQj24Zq3gD/70jYgGpy8RfA0BihSR6F+A0WZLh/fESjqfor3/pFWhUCRnPEqrZsXyuB8lRgDTyGp4s1cHyUqmFMuCPDRMtok0rSVpaRZj2Zj8lboah19UI/L89ptQgEdqsOKzlkxiT+1GdyIY+jpPCM0a3yZz75l3Qiu0k6q7qUV8T0hm1GZiBguvST1KMcwnt kyrylo.shatskyy@suse.com", "key_options": null, "keyfile": "/home/kyr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kyr", "validate_certs": true} 2021-02-12T17:50:43.978 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDeERrQ0NVXhuTYzI08rlKRbHZVFXSv1YC7sGq6aCWRbTCv7Ad/t/3OJz3vfcd2TjBMvABfrYCFgTFWIaJBaeA3BXUUU9y9TwnzuxKv4C46WJYoalVGyLEpWpcGbJ5v+leA6Je/+lunenplb7TBKGE/YTgggEXyX7KnunYlv+Jdm33TmAZTcHrLk5QR/6FdRGKmdlLna0dXZc3v04+AI8W+p9ghAv1+e8ENxJDTF8p0MawBiKWcSj2mysOhQ2Cj91qZ0qTnLLeu0CPf4rBO+qjYIshiCuTLhYt2UazlzME4nCCsVqK7Nz9olA/lgvJLRaPxgQQ7iCMmIZilS6B4Y3oL sseshasa@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCo0zniUEXuQ4JJuuRtfnEK8WHqr97AmXEniGPY5U23MTxhMuP8CSkGWkUHdmXw0H1tDKbYgk97M6TspJropNGKljla2H1/3e2T1BOI1L2WfcsOh6VPYeyOj2ssMJeEDdQgd0J3g+4oj2QS+guPlT24UhO0CBhRZ6waVlJMgagDYyDLCLGI4ASoBYx0AOaoyjnEQtRd/gyejzv7jntkHc9YNBPsJ37X6hTYNA/s2ORJE7QMXd6xI60mTBuCdD3JcPYoUSvyCUa7rIrDc70lAcW5XtxewvMBogTZJLoFG/d6Ugij49h4AfwrtoUCtLyJOWJX6PTfomXely8Ym6KpN/3981+m70dhcHPDJ66QCWMd1LAgftVXTIsebA+Bxfg4Vsbrv7j1lJXzmMfsOl+1wVPe+Y9h28GyeIlalW3laURZGV7dhexGZ8YNxPuzxtAlWtfw+kddbjw0j4ODtgVXrj4SDxqmTTYYuMBkrQ3H33sOgPLx8DyA6e4pckb6lOOEvb41Yi5tyP7BIJlYN2PpNVDkSQd/t5Q0di4Ct+3vqLRCkPuJiJAUyaBGT4HM7yQc25+Y0B3sXNON7BjeaBXB7OwKZMp9w1jwbJt4JL/aGOw3HVljVyVgu+pIuQ2ISd9VBcmAJK5EGdWe5Q57h5fJkD+TVrXgeNTaufdclh5vWOH1qQ== sseshasa@teuthology", "key_options": null, "keyfile": "/home/sseshasa/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sseshasa", "validate_certs": true} 2021-02-12T17:50:44.185 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDIC6uqbIg9XaTlDZd7KCX7FXSzfGFMEWXhx42ktz3ehOPLviA/yNpVd+LmYHSgzd5Szd3mMxleisiE+JdSq/q5YyPAF6eD6+wfynkpMv5aWNHGI0yVfKbCwdOX8bY6yqhyVwMDo14uoFvdDcM4Lj+vz8F75VJHUU5ihJNOW24EllobTLzR4XI6AcM+mjbXRnBLLVoBSuDaU2D0kWqMn8rDCJKdUJnoiy9oxIok8eMLfGkGsL55oOx9BflMSz/0jM9X5GLQahYiECbXjvCIarCkUFE8txNoXM9Djmg8QqhAQDhY0hzJTauFJJaD8Ax1uFQlXONed7tv7a7Tb3dH+/jb rfriedma@dhcp-4-239.tlv.redhat.com", "key_options": null, "keyfile": "/home/rfriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rfriedma", "validate_certs": true} 2021-02-12T17:50:44.392 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDIXHA8BwV4OnqlaHHyYofsp5X+wanose+aneJVx2Or05PcWA9c7xFEOerkvEn9LvivSRd9G9o/S8SvnGH91tG2bbOAidVZ69FGul9OKa/BCJYUP0vVBX09/Td3n7+R2lVlvu5XiyC5SUzCjaQEZLOpjq23rqaqdCDXE6wFnxfpkmcsccbG/xUiOEBeBGI92lntdx99Sn6CxWWA27lMMaYkCRvX6ziRZEm6O44N2nMm+qYm0jmMyGv1iyQkJxtt0YaLPp+X9t2trUEzEjgBCXqqzEzBjB7T4rbzhAuvU3NRHiAoWXSnrY5AQOTTvC/wpaZV/gF2NDNThjMcpcEdNstleOctXtN4XTTJQewbG+t7I9knR00TPpsoUvUtpbcoFOluGgWTDTFa6GCmDPppx23nZgulOniZUVTI2CbScSpc8sfp1JE7Q6vLECY9lJ+GpVCbDAbIEE9WHXSkJ/6L9yf+ZoHbpqCkEFDcWLfz3B5NFoJ5gne8B8uzd6ptKVtJrP8pMvoOyydyvQlmkgh8tNDsOnp6kaUNG0NHmtfGlCTkbM7oKsKbf87Cv1HgYuZl/HP7go5sOX02LfBfbvWrawZzOaZ3dewEq74w5hVMKEHrmJPstIpcHSxZpq33f2j4Z/uKW9tJat6rgSWK1I+T3tCd++76apxbOYl5TDOaoxCyjQ== lars@marowsky-bree.de", "key_options": null, "keyfile": "/home/lmb/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "lmb", "validate_certs": true} 2021-02-12T17:50:44.600 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDMDUcToIhs8tiIozgPudiYyD2upEX+VvsiJRti7zrA/1HPIDARYt2+xyWCASm5x5vdohRM7wnBLUlbBVnvr6ZUQvfShNDsS1Uu7hEfYF38MtW+dVwvGlm14J4m1gbjzqVlZvO6bg6rfGo7Qer0bKUlxtqbisQ/kCg/lVWR04CNxdRtZxEpaQ21y4C5jAlrpFTzxQSuiCLEyY3EAKdZUaZElFEcXlna0ysCP8inlzhF402PkSdl9DkkPpg3IR2dUEGqsblWSdLVk19EFavjtGAAnsvaBWWm2rCLNC5RmhGQaS/Z52Aes1nBuT4cI6qLjbEMqAzLwxiOR7l4IPMpK5EcuRK+FaqQG0wMAjxPc95CmxDzi58ZRJ5ERV8vgps73F1AJXWU9Lb1P77BTsLXqiwko2mvPf6e0LwCxQuxYJWc+xvO7Ji011B+nm5eLCCC/LcDtTPjkimTh7nkvUNi/yTKCXi9KwLISAHHd335KnfrL28rU3GahMXs4NjOxuN8eKM= ksirivad@ksirivad\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDS3HkgHb6z+gi4tN5rBL/N+WLJgN8RtUokt0YnRjpor2epRbmqgmEFRbRFqjLK9LiTuLxmkwol1/G56AJms9ixue33M6jVs73nc259LDxn2qj/NCtxeU7xdjSDuhKyItgM0yuiFjWOKSYTghNtwoR26PX7SQVbycD4NN1ZuwUbdMQA31VrLvQR7d0lYKlis2auH2Xb6WifotoEyszZz1KYRwzJvOHtHku/jAyNmTD/8kAsuCjEgUPSyyehAfdWn59tY2gM7aFyj/dtRoAghekv5gkov9zx+zjN6D2LSN7e+DRti05iJMifA3nfmdj7f8aHzXDtsDL0XNkmmT3g9N2OQ8WL4ESOFASXPPj9JGUROealgnaQ+Y64evDClhbt2L7FLbgj5sgKb+XYhpGnAiC+v3t1Rq3HZrstOYf2qSYguIo+zW9gHwzJDqH5YNCpSF4ZDCfkJ94ngl5ptQmZ/FcrH2hBs2cMoyK9EyDpcI9hBeACTzu19kaB6DrOC+UTvBgYL2sNbCvi+caIqhENJICZ21rI4u4EZqu7PVb+Y3CLgSjTEgclGYBZoxlrT/X2QHrIfZF7lLGeeh0bureedCrBg4b61SL7ZU3Me9x1iWOZMSJgVvg+qPSrzZiHbA46k8i0af5Q+geL1BuuGy76zbYd/KR0KiNZnXjdlHyRz76rpQ== kamoltatsirivadhna@Ksirivad-MacBook-Pro.loc 2021-02-12T17:50:44.601 INFO:teuthology.task.ansible.out:al", "key_options": null, "keyfile": "/home/ksirivad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ksirivad", "validate_certs": true} 2021-02-12T17:50:44.806 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC0POsA+0NyxecLo3Y821G6W3+jJPKFMuLSIl4FSLi25vuyuM5+mFGmCNWv8gg93eQhG9Y3SOygqmkaxzxDtBS/V6VNC2f8hZsVw9B+HbnfggvCNQSgjJ5rhvf6IHfx/aHq4IHF+stqdcoWhwR/68wIhGNqtP7Nhqnzk2aN1kG9bU2iVk4CH8XGIwqwET3aWNuCMcHi2AhrvDBpZu7bzo/6ARUJDqEfTEC/XIkzv8/mEkJ5igcycARVy6dChuCwfZc8rhfwQpekwx1Y9/WBFdNQlYkN4fVe8yIqT5VsQ3AcRv21GJk8L7mwB4hF9dVAi4p1cSyxatR3kDifsFc8VQk9 toor@localhost.localdomain", "key_options": null, "keyfile": "/home/ofriedma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ofriedma", "validate_certs": true} 2021-02-12T17:50:45.024 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDc/8sWrMq+bqKBPmDPplUgdMoY2SQL+57jRSh27PAoYW9rNVR/UgK7EA9oxFLfoi9wJ9+VKU4opeoohOXLWQXXatLvfXjGE3fRYIonQfwJqKrasqQJTGWzRZ9t1/2Cb0VHkoAvipCJzWmgB46X6q73XW5Td1H3b+AINd6E4Rsm1QjBOhk0DMjCdfwrVUZIfGdsEo+5u/AwioHwH+NDJvXTWT8ZigeZBNHJSY/EQn1+15ut0QB2rvXkJasvvDlVbEsPQtM4C3VX1ronmAnKMvCp1O4EtAXaoKSOx7XEN++UAvVn7czDCWk/UFgpzZOqTcQJwg82XtS4sZP23CpNe0G9 h4iresis@fedora", "key_options": null, "keyfile": "/home/h4iresis/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "h4iresis", "validate_certs": true} 2021-02-12T17:50:45.230 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCqhWVTosH12864VGY5S3/k0yKc59GbV0m2WxD28gAf+uvJ1/y8gekD2wDzBv0DWqEekYLV7L68gpapDXyGosQeZ6n5OO/NZFZmW+Xq1P7P1QHJil6kVpwk927lzX440pwNf3+PRvTbLzlbuVFWJRpiykv+juJyIFjzXLJRF67wB2qgZVm5cRs/GOi4fnvTaou/Ae51mvsO8M3JL5cphApFKD/AO/X+3pHBhLjvaW33cYsp0Z4Rce1nm8wxSmvuJRF+HAfjBlknhDl5ZaJGA96RYA+g9WbAXimFprg4xeiRp7liPExQwtVgackJweoC4MahC1glmjC3eOh9dvrrxLnALOLMOkyCSK/tOF/iq2PJaLViObanPI8mnNkMLbdJyzUBf8hRGdSLcw98qxaf0sSZ+vz4VQS6e9CwX7ikkNNsTeYXjnJ+1yU3mfCW1CXJDr7kBDVG2OTLefCpMY8WBAYDiEaK0RW83cBcFIKfecKYd0qStHRaOsvmyWdFYgBh2vM1UWjro5WXlb+7aJPrUp8ZruapkUXUwhjHA7FwGA4ysE0tsOVVFm0Dw8tvyVVkVNm/9EObG28dWctioZ2+ZEzTWgMJU26Q+wOJ/U4qZyIvpMRUgPZc5nM9J8kcmvNWtb6DV9k/6p8h28kiOxPkxLtPP+3TlBCHjq//mhEnJZ/d8w== shshyukriev@suse.com", "key_options": null, "keyfile": "/home/shyukri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyukri", "validate_certs": true} 2021-02-12T17:50:45.437 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCxDJtmmcnrbA4AUsmBBZq/M4xNdIbOT+cQQxhUKeCu8F9mht+RznwyaaJbVIlD+UNv9LMhRyzI5oktSalokNltlrtPQMoP52PXH7KFFGnVnexCHAk4nGvEhuZCniO5T+ENm2wFR47lFEQK7lyJJLp5PFiSM7vM1MK6UkMvRmCTRildjaYLyf3EEvBXifbsGmR0tBIXtml6KITCAaT6l0E94055oEGN2G5NhZIvfLcojt5liEkXEHQNdfkUDXM0Go43JZzgsSWDXC/7RYLrAJwz5mq4qCa9YZL/3Spjso37+1oLq7lOQ0TDJUAQXmnW0So/NkzECx4MolVcylhTzmgj sergio@milestones", "key_options": null, "keyfile": "/home/scarvalhojr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "scarvalhojr", "validate_certs": true} 2021-02-12T17:50:45.644 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDEOigBGMeElb3uHXX36HxLtPJMG+iG/6D7NDFCVR6AyYsmimBQb6SlSIR6SaFULb6HT/IFVk0Jf/z0HoipAP7Xu+wvP/J6LFoMdguxylh1ov8raqxQJrGLl9n64EI+qWINwEM0MKMUnDQ3tmMKBl78LKQVQTYBx1+9cXsT6DE1yhgmV79XIvOOb+v53agSi/kLJF2i9K04kpi1OejXg95Ub2eLMY69Jm8PnLy1Jq8DhC8pz7nrfzbKtOC7HbytTAO1XTlb90lVhqC/1yTMAUc/rpJwoSEHc2epGeZWdMsJhqhX2EqMwcV53M3qcFSGreGNJ/Ncd4/JHxz7q0IU411t rakesh@localhost.localdomain", "key_options": null, "keyfile": "/home/rakesh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rakesh", "validate_certs": true} 2021-02-12T17:50:45.850 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCvRwrsN62CReT0/kOeFEXGA3THhzJ+hwZVpUzo7VMRZeSo6LgNlVGdrQchMHnlwBvuwaVNJlBAkQr+5J5BB3ikrmbcnLHfPutXBBBB6uDfSwwbqjzUY7elTbz9ul6AkegR9K65O44ouD9+WC76D+Tp0maqnP1w0O3h1GVxDHe34LpEwBvZJ+/BSYmBpwRZqeXFRADORPzdtW08PSiRErGHb8j+j5i+br37JfOOh+A/9dHwQw9alBVL2DijwvrPMa7dV5yX00dlMSTL2LLo9IdI4aFPmie8oic0eanFbQDq210ilOR65IGeshKHEKOq0KFFz0nSK9OdO03OkSEN/0d5 ryg@tapir", "key_options": null, "keyfile": "/home/ryg-/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ryg-", "validate_certs": true} 2021-02-12T17:50:46.057 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC0zCoXv2Kkxjqv5FOlJZ2PRA9tStdXAVGZJbSLOhAWeqHQL6O2l8MO5GFzHz4FA/g7lbNzLKaCbG1wYGuL9FOt0o36/eHykgENu00SlFunIQn+T/WNaGxrZqnfbdS9TdImFAnVY4vuzrBhFTb6qcqTLhw1WVz+l/iOv5CoY+7NnILWcj8dHNcPzU1ihlFAbvQgQcMtt1130zqc6L8o+SDm2YQF2nuOWxj0LbD6YpRn3AqWsTN5QbqzdeaJiUt6kdywisPdoiSWLuivRKhNLNhnWO+B4DnchwfApy8ovJShHZAsrSoc+a+RQDmOiLXmc5wAvNnREHbqod8/0bQ9vSXaVQCt1gnnK8Bf7GQElg3Z3aep4D4w6rBB+LBsHFbTfrzWAHkXxJeGiQTlJGz0IYHi1weWrzY6x5rEA0gYDu0FVLOLrsYDtuTQpjcTwpmaHCsxz6wU3wCNmTsV/AZhniFHJhHXacCmDp1JfsI2kP9FjcJJE0SygnZ/SzKe0DHamnyT1OgdQaG/KX2eRv/vXUPg2ukKiA1fyFnFeEsWK6rUO0YgCswqjrM4vU0leZ1b8vK9aM4IDJM3TJqdMLfOa0rPntekNOqXUn213CP+Yiz9l1TH4zX3P/Kv6Vv1gTeGcX0lMaYhowng+97j3LLqGcSUPMklrKen/xrsACFxN9fxaQ== prsrivas@redhat.com\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCks+XmOepYEdSBP6U0T+r+fZowSS7UHY5W8AUtAkvXiKlTG518luG2wtSALIsB5+jyjC44ITwV9X/vvsDaXQnp1hFanwWeuU1jDdy+MvywvxI9q4naWlDpxrfq4i/AioSvdoIo02uUFYPKTCHrm7wBjTwpWbCD0gbC9/OGAHM88Md3F0qbgFEr3BERvNGe0coCjWRMS8+GTZXS22WW1/NxNzpdCUjO/9j4QydpW7UF1kqVhnPVZozmh4wl0rk8tgb9Zx3vCUBG1Q5eO/q+HsW54eXLa9FcuQcoJnNdV2hMXRHHZHTgaPKZ/ildukzZHo0i/Dq1mi5N3s5A7aDYxOWWlRXazwL8zu/UQVrxVRni8vDTFk5682lskF1WifzADaqicUWzasdK0PC4G7JmHPi+XHskeOt5i4ywEnFRXnYtMOIu0CNO82Rg4CdUAD58YXCGoQ4r9TG4LlwKg75ouNHB/ToJidSg9NBwclP+MICcTpWItxVXpBj/YkeLcntHtVTFNHORxIwIwWng94wVEDCfyXelyAR6VfwcPlWWrbzhbEO9DH40GuluQi38ySWi/HpnrVriJMLzEQA9OPrLsME1LDgPGkF2GloPL8nnnLWB9L+YEBNNAVoSo0PHBsjIibdZ/SZm7zUgYGp4fuG+m1+SDOjcfD6GSXOTQwQou+knEQ== prsrivas@redhat.com", "key_options": null, "keyfile": "/home/prsrivas/.ssh/authorized_keys", 2021-02-12T17:50:46.059 INFO:teuthology.task.ansible.out:"manage_dir": true, "path": null, "state": "present", "user": "prsrivas", "validate_certs": true} 2021-02-12T17:50:46.264 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDOJg56HsyNa5vHHb52G6O5vtv9I2tdlNMjFlG3SVCCu/bVZwVeGzp1ekuNGU10owmnIvhlSldMjLn2wwTwiDjbvYJVVCPvzK63vCUcMevRsvr9EyU16JdAgrmrGMyuYYEb+wh8XWI9Dc72o84fMaBYFSSer1S9nUBL+KVLVhIe7culgADimrDgDVh2t7u2JHBI0m0nKFGJZNcIyYf+GKOM+n7SaQXIYIytvSteU5BvUIKPsDd4z5lVlWiKO3kG6jF2RnKFV6KWBWI9m8YGItdGu25mXELXN9MOlApZF3AoGTmCE2Web7uDVGvO6rz6b+dc7RXTOlbWIhtMoRqQf6mz zac.dover@gmail.com", "key_options": null, "keyfile": "/home/zdover/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "zdover", "validate_certs": true} 2021-02-12T17:50:46.471 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC4zwWf+kMEDgaSOrX7AlD+UCOgf4/fRiorQVJqC8GfMNynfiFVZaThngwIrwtQROfXHbzspi+AcRvDLewY25mXyuxVgOfSbuSIEqiUNvzcVq9IItI1J58xeDBpIktAsoHHS6JENV56fAT6krzRAQLnSa1k2b0pyKuPBzFSeNgiaQMhjrCLAaCi9yLsdKpnWgXhyfrdqvN2r/Ctts/zlTPqtRYdvdIL/p/l+vg7Q8S3OhuAGtimaEq3RE70xadaPbGhsLISMTrbfYXbxw84KKZpNYwO4zmQ2ztcvC9D7A/8ov3+A5Jq42G+pq7ORuix2NoTWtADMiwvNe1HyzS/7cX/BuXLFssRY1bzRvp+x2ExZ7b+bFx+TE/RduRh0eyDNLg7SyL83aeTr2Usa+mPoCAMKed9XtYbRnUu7ox4wmfeAgx6jM9uG5uQfauJ1XpDddrVBG8KYBvSd0BM2cyRdcAXqwxg9AVPZcdE7sbBk44CGVhw+ziZ3e3cLBTO4y44KRQre+1Bhe/Ze8OTQWLFugl/XagFlJE/L6fGUIPuFojmQvlw5dBZwtUURnxsu8yHm3iIsj4y8nR1zHMj+zuMk6rF3ZW0epPAECAdq4gG8X2wv3HTq0vLBkKpi1KMAmntEtcVwF0fzMt5XYqRd4jceFgonAAipJNDknHgi6yaUzBYmQ== rraja@redhat.com", "key_options": null, "keyfile": "/home/rraja/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "rraja", "validate_certs": true} 2021-02-12T17:50:46.688 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC2mEyxfkW1u84u5BocLCAXESj2qObWkxUd20cmPolhJYlIWLOafa69Dw5A9CjFBp0U6Jx5d0JWcwmAc5ZeffQ7t7vzmtHCZnpKkyZhN9bu6vuCh4N+I643SI/K7tdbB2E8ZE6uI7ORMAan5+t0bNSVolUVq6X1w7dIA5El54K10WNdiwgRKW6e3anNbJ3rBLGEk7gawZqDDOqTvVQmOBGWek3BjSReDvg3rkmTQLxJkiKdxnx4/s7gzH1xTpsS/OludIuSxF0BK064ZcT1t09qcjnT4Os1inV2GfYswt+UhpgqrHqXjEs0Z3y4Oy77QBKL1rDf93CP4a4+eJjqXe+v tbechtold@suse.com", "key_options": null, "keyfile": "/home/toabctl/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "toabctl", "validate_certs": true} 2021-02-12T17:50:46.895 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCitWIN7jm5KEmB6jqcA+ufKRLodxkJ1GIdaaueCdeXdMHgWNH9FUDCUJk5dcA11MxQO15DRH/cdhb3QHPGiMv4XrBeK6HHYwGttzkcUEz1EDUhpowOrnFETbPa7K39ObmRcgdvoGXOSrNZ4mvptCTz9Xb0MF+Gcg5iJ4g+qZzIVa2/ueecYY8ZorbaH4m/NWUub6Rf01Q04HfxTvKrUFswrbCoE/Aigz/Oh88RiS854XoTvV7rE1tKbyKkaungwMAaQSNnZkq4hi8nbp6pzTxJLV7N/ZYMhVpdxSbWcghFg6cy7aeoCzZMN97WcDhpdfbhRT0dbqCkMmHV6t0usDDnOss2v68UCzkxuA1Hj4uy8rRHjXBtYqI6xsrwCjvmgP05B14TpmFdG+3GdRHx2+QlhnGj7nuPf1KoWW/f3tD85DXoEmVl6RJ4KTrMeEVlOn3kdMLVH0y2hIqPTYMP5cMVNi0EdfeoBdd9kxMKMvoSVBAgmLzGVv17o1lNvVS6V1kyt7gME2PfNR2Np+U1cJVJfUJRTm28mHXBVoDXG9s0zv8+LpQ/gVPS+vsPAfxWWYGW8ZmEV3ZorDOZycSPybjHkPFzDAdUhPG0Db0o+F9ug+22F7pvifsdDZ42zqGGLbyzfIQE3AtyI7LR7Sc5tpN9rKTeSBYJ0yNcDKEj5FUHDw== mfritch@suse.com", "key_options": null, "keyfile": "/home/mgfritch/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mgfritch", "validate_certs": true} 2021-02-12T17:50:47.102 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC5bCXQ/B16AKl2oevC93R0U+3OrTyd3XEypouJy4LjIzXLX/Go1C1ViERwDAeHmqiUTqJUL+KZ0kMo8U7zlhNBk9PucbMZCqdqx5hkiCAo4y65lEXmuPGimRU2mK9TpcDTLWoyrD/EEhwzT6gmw3ytVTxi7qAb0tQUCQB+pW4+Zr5u6HZzgu9kLGxp7zsGtMA84Zqm5ql72mGuS0bUN1ajm8LWBetFuzOOW6YGfnKt29cLV2h1itVewA9DKM5G70B91eK4LjoNddCkqVA8L+v6LMQNU2pejUzPap98hmFUpQMNxlWQ0+B7UjfbkHWe+mIsSkdr3FfCfqWvsL0RiHcZ root@lxbceph0", "key_options": null, "keyfile": "/home/xiubli/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xiubli", "validate_certs": true} 2021-02-12T17:50:47.309 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDT+6B5Fr99p8xe9Zo5x8ULcdeA6ne727iVJp/xaguGwGHj6AhuK+R/CjznnHk0yA+6kiblNzjvbnsuzO8T9c8Ayv5KU07MxP1y1pymdYBpEh3NTXmhB6QOTCNTQbT14PL6MVgzK17SqarFCYXoEGuyK9jaBdN9QM76vhEaXc/2Osg+d8boJTkOk7lOPHbzeOCbQU3d+jX0VFOgAQS5OFz3Ug/X7nB0D0LN8bt5FlCZiGO95Iwls/SyPDbVXc9ot8oh4ME7kd7sElokBdYdXPauibRxWKjSDn+hV6kZfSzqwnMFaZ1TC+Av6zNsBuZucNSp3/Ws3T5i9i+21eyS4+qVE8IiX2aVKYuC9afb2lPLbdG2Iy1CqiaeGPkZfiGT9P8iQPOkqae45859UzFAd/XzKjBcmus3lSArKG1EyZl03HQ5P2J60pErCXq8W9mPjxAze2ReLRXJVEgTXnmdhCptcg6PZ4d/k4BEddbdnoX6/BEgFxLEJ64HpTeTVcDXHpipCpxFSRvgyPH/wQGKAvt8AfKBhmeZs7LuyGOhJiOon4DDaShHpE9wTnJ5qlnyg44+u6NCwelGPMAkHKEO4OgcjSsyXm+HYUpxG2s4poloIJzvU6oa3dvJsMZFhcb8CsW75sBYE+YngS/1fg1WU88ZOdaa1yF4F87pVZycpZqY1w== jan@jfsuselaptop", "key_options": null, "keyfile": "/home/jafaj/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jafaj", "validate_certs": true} 2021-02-12T17:50:47.526 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDo8wE/b2/EDwV018ENfcENVfXSF7vnXAwBSzNVScgxdqr2DVKTWDzhoI0m5CXTqI2HLnJVFDTgN3t5/fFcmwmSAP8ic8wU1JznSPZXN7qEQrcEsU/cwHeuzyutibg1ItTWWtoXYV8SFnVom8NF3UwXF8I+Vj6VF1EStWCcClgPjLbiOm8vIgaouXtbUIFv25GpadXyUtwiHEq1NKUPLgka8Usm8ibyS9hY9KmNWlD9AfmnwAImc8Urn2dErcsfAltxoGUHCI7ZBHGpzg+6dpynNR+giJMrjwWzREaCHmiJX3g6jmHuuY54eSOs6cxqi3E4jNHVWXPe/tEF3l0klEvl jschmid@suse.com", "key_options": null, "keyfile": "/home/jschmid/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jschmid", "validate_certs": true} 2021-02-12T17:50:47.733 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDnQfkkxYWLhMdoQ4RsZxWTXBpbzsssgeRfvV5q+n0mW60dxQ+9rjDXUTsRkmh45B5EhNWlKwyIUcQ5HpgJk6XQvYU6MPboWjriCTngmoUbwNIjOjyazkBbxfbwUiRDb+x1EyVm4Kt0rgj3nhqqxf3krqO2SlKgsVaY9b7+NTfshXb9T73/RRLtvwUOFkjp2B+w0SQs7Ka1GenmHSY/6xP25Mb8q9QW/2nuf1mV9rTd5l2ZHREzgG7gQFhjgUq3w6zCCfcCGwrPb4omqd9xiYLDw8l1vfQ7UPbuOqK1rFM67r7JQDG3yyJjCdWgRUI/ysbbXj0fyT+QOPZGK618V8yZPVlq5loQzYI39zAWZU9L9X9bkQCsJfjU5Yo9f/zOaecnWfUxC84NKpbG7k6XACdUIB9JGGtL6RPtgBU+k6Ts7EemIUCyMYm/GX+PYU3GnmmdqW9yBeyZoldwDCWLq05vjLXj+R3J2UrMl5x+fhP+KEFd1ZgNc7PV+CF35/ux8UE= kachau@localhost.localdomain", "key_options": null, "keyfile": "/home/kachau/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kachau", "validate_certs": true} 2021-02-12T17:50:47.940 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQujcXnq9qOCmTWs44GwKOw48BxZGBmo9AeFk3KPRx33Cz5hS7UHPYvJy03KdoMySgxJtuxsOSvjJSPxpXesTtArO5giW8RLyuVsu9q5j/GhPKyABttuGQyexiRokrHQFLVifzgqwsUARwHarWH16Oa1n6fZPFNqSH56c872zS4Pwqgkzx99NKRWh/B+fIk8VmDzzP1qvQAnUXDeTeTOzapAL+8fSNTp3QnhmZbvCHYwUxfSqJyzq1wBL+517WrvzvX7yCY9B0wzm72sIO4daV6UDyAL1B72QpB04vEZZ/skwS/0jmokfBn43HHCMPy/Mxywfi54alFJzIu4pti30V kravishankar@localhost.localdomain", "key_options": null, "keyfile": "/home/khiremat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "khiremat", "validate_certs": true} 2021-02-12T17:50:48.147 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCoqsDnRqAeNPi5xiIfFl2i2zqcwLTvcAso3R9+QAfSChw4+wcQzmMV8uQz1jt8UReefg2QL6bwvcQPj5Zw7ADqUjfxm1K0ezgH6RrtHgRZ1SVseTvJUvzmkDZ2gr2ToZAJYsaV7gBLJLZ7TOiQ6jzrtitiKqJZdpZfzadjShecY1TdMo8bSS7XfbRcwri40u82TRM825B4L//QuhMI7akPqa3iJbGEAQPhkGLI3kz19ryVG1UwvAkkt7TQ68zz+piXSmB2CMRti51shT4NuAOq6Qe9y2Obicf88ItkYxvySMQ5jWi2bo7Zhwuq9jUH8lK6I6PHrYc8LOsR9gE/hqkXxTQrkOXEDtHN1mr/KKLrXO0GeBdgh8Gj4gcwgd1RUCE+T9q09WO9j7c0piWK9Fs1+q1FjH06/AfmP4qob1TlJMHhtPyOp+lk2XvABP+sqW4vyrgmXu5vJU+Hgq3X9KE7bKh4WkCem24Np4uA+irhpvZmIoHyINsYW46lGfwJvM8= kapandya@localhost.localdomain", "key_options": null, "keyfile": "/home/kapandya/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kapandya", "validate_certs": true} 2021-02-12T17:50:48.354 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDNEEz49M5KdLcp0cbKrE9iihrH5PEhsPgnAxac+aQwvnTstOOlUOyR+TvHiRa7sgXHuFX/UXo+k3Qmhs5U/sMZWZmmN3Jb3fKT78lSBbGnByclfPoGVTTLOHxwo+RFE79Sp3e30oshJ3DwqYgiX6JjM8B2Y7d5Tv0Ss5uEZswvmCEjdTLDSsoxrsOjVxBpuArOfCxes8PnKDdq21mWldpP66iwFel2GgOGSc85iVHKcOE2OpVMazMzXoMSMFoOpCULKjFfXBuDIqkXwJmtLJaiCgcVmTR56KXh7JpZ7vAeJ9ttv9qgOckqKdI3jEjeDT+TrdLlRWghBE7FRkm0vmun prasad.krishnan@macprasad", "key_options": null, "keyfile": "/home/prasad/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prasad", "validate_certs": true} 2021-02-12T17:50:48.560 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCu9CW99ySPXSKQpLbefu1A0KqbUQIzvUOJizIu4azEgdlOqUnOMR/GC6jgOdiVL5mNb/U5bg7yQuEHgPAtmPRotvbI6/FSwIAR9+GgiuKSOJCa4s7QzWbHwv9g7wS2SLURw8UERSzlLlT8KQyGj+GkBhToWp4PTXC7te61c8XJNGF6giLmIAky4SH8uPI1zeLwwviJQomjFWZZ1RTbcRJxvGPjPtuUKDlF7QgXTLE9ZgsNopusC6PfRd7N8Udkl2OYPAzA+Jun0IRnQyjRXG3c2sYXQR8YYGICsxvnm4l81vKdxRL34K7LqBUHQXUcoCjJy7+/2/1dca/AMHgi4sZwgeYQ/U9jZg5jbm7o4MpDUFsS8W6OO/ZPiuYTswGBNRXhPmbGFnfa9dBXBS0p+buHF8PcTSLH0Dp2nhHkK1ehGmBFVPw6nPj3wXA9BKuNc+g7AiNUyRKyCWroFJ0VwUQGbI6ZMccO2cI7QpYmIqPCl3am/vUSNq4j0m/iURR7+t0= vikhyat@redhat", "key_options": null, "keyfile": "/home/vikhyat/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vikhyat", "validate_certs": true} 2021-02-12T17:50:48.767 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCipD0d1tI/Cff3AeOZTQ9eb6pZ/NOCAOw7MZG/JgpCs8+T3WqiUEYefsb2PuSa85vV2GC7kq0M9ux7+U+vZZf05geYiU2S9eTucLEu2+YE+RaAvIK+ZB134/33JSPGIVubXgGWZw/ZFCEYXmYk3kyEg1YDIk/BEaoG7PsDiSB4j0W7/WiV2bEEdSlTXt5C1/kH0qn4H9wQMiBSpdMgdaRWw60HjObDd6LASeUVgabbyclypravZdpYBnits4ayoN8VG1aZe4Qhm1SzP/CfxAAYfxo9NbqSBWJmn3iSN8UKHhCFrRCTaYic3z3usr8784VsA7z/LxK0fs6xjhoxRTa3 shraddha.agrawal000@gmail.com", "key_options": null, "keyfile": "/home/shraddhaag/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shraddhaag", "validate_certs": true} 2021-02-12T17:50:48.974 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCpE5V9JG1WhkHSYqjqaijBb1MLyjcKQlmj5WQ8w5832n6x9J4yDO8wI4qmiX3XeiXPlmW6XjAq5keyukrQe/yVI6NDXzgRxaHDmNj1AR73gyUVqRXpJprrqqsnLHawIRovrBD/HNhTv/2G8xG3BHakac1+qj7tZe4Jjx0aG1HbRhhJ5Q98EgvTkDAhktnlvF9OXXC1WI7QirHx0pRHa40SYyFLfOQDbQSCzb1TAAY5NGn3SwK3s2mWeoEsAJpCLrkNfSTbR3Z0mylAJUn+PayqhJegjwKmLaONjx8ZHXN970JHgojBBGOD30J/NnJQ7N6eRl+ekjpHmLAYGbe2oCwD gkyratsas@suse.com", "key_options": null, "keyfile": "/home/gkyratsas/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "gkyratsas", "validate_certs": true} 2021-02-12T17:50:49.181 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC9sZe9p48s6RSpluK1R97X9tsN3tWiidPbdyXvyMUXifYQb3nth9wMWUakrVBp+QGEkqALLIf3so3MfgbRvYexzNfHvRT7Lxn8D/isiCpWa4aLM/0alprV+RI38dR0mI3IVb3aIEYmgp2IY4yzybWTHCmS1VUrUJJ0pOfXOssq0xgo3qeaSQBHYRKasqR8GWAD5MWpG6/kg6gxI7o0aHVVTXy5ADB7NS1LEZr06xxNm4exeYvl9TzXrbGZWf2gC8ZpXhmFHI80A+TKEZRU6wmcju7G69nyEEF8/ebcBKmZB4eTsDN7qR7lc9dxKxiiZcJXc/MWy4bSu69NhOeuq2xx7bdexS8OHKdgYH9gsJolu5zgN20IHzaRgswO3XIe3nh2Ud8wntwBCwmJ9bf5YByITGjpOhntkvWAP3AXkQ1B4HXCdlUNW+4pMexDe2D05LHOwxcS8RVIH6ELu8rWe/QBA+DFcyZz4xE55pbfVt3FCEFrAxUn5qkyzszwPMjtK4XAnn9kmnw2B0k6XLZOZd+n7W8ege6wih+gwfDhYMZX0qzi3HdAK7nk5vEOFibCztLaogzfc6fdFRx8OIWJQXqx5I1641FU3UE6sVxku4IMDTq2GVyUWKdSN6y8gBiDPVe3xu4qM4xghSnaAmfB45IrcNvibBjikNva2FHt8lcutQ== prateek@prateek-Dell", "key_options": null, "keyfile": "/home/prateek/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "prateek", "validate_certs": true} 2021-02-12T17:50:49.388 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDFmmTp8+YDXU27xnSPxhWwouslpkvp1eULVp63taoXceDOnH8EMWv0wiEjN/P+Fcf/UB+iU8bELyHubZiqnOUr9f2pGX4jkJ6Yty1p5BUK4my7Si5J0/JawEDVQm4HpFWMKUVKYumvz3UFuBoINHVTqr1cLIDAt7mdeKfJqFB45AqxDUIPP4d8iznNCw+QO2CMxk0o/wMNHbygZZrHAiDngXQxwhe6oR17RfolhREy0JjHgojRjxgagpG7O8WW3OPI5Outnz5Pvraas5q7w+b3xbB9vdTObW/Nii5VOpTdIsFzBvp46hLoRSIcB8pRLUrxPJBPOY7G0zbopDRLB2zLNGKIH2zxEHJOeY928uUbWT22b9mGu/L95FFY7hmeRAicCUeixMTgXgczeQlHDn5eqQzLI3PNP7JX/0dauCD27GRKsDUnpIDj9RqgCQDVueIQQMC6QMa9dYQmps0r9Ljp8R84WjYFRP9oOI+POOzbgmRO0JCVnPqz9MKyGVby9/WFBHcvG6xmwACez00oqe62z+rhcavGXViX5CxGMV6vUTXugBa4ZkXlzs+djzYWfZ33ZKVeRIwj2Pu1PRTGzB8e2BaKqJMN0FVtzpV4hom4Y/9vIPyDpjRjNuKzrkYW1M8PxBxn+EJUPMLhr3jFMCo6HSxHoFXy8BzZoFkZWqap3w== JCL@IT-MINI-JCL.local", "key_options": null, "keyfile": "/home/jelopez/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jelopez", "validate_certs": true} 2021-02-12T17:50:49.594 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdbJDYY43Zu0EKAGgrSKEobyXIxdhT0AeTT3yBoNY+YdYDScimJsQzK4hKNJIn1xC3klr0dSnsqeJru+SA4GMhbdcsmVyudhutCw75R/vFoRHOwaeVS2SMiBAtDbAYGQ3dh6LapkmXeLURcycNbebUV4HM9XsYqnuL8J2XyY4tRKgHrAd6OQw4jsXDOxwTRlHM822N/2kgv8iEwiXAJoUoymbr8mnfSxuuaH0oiKBwuaK1wuaDrlJyXkXNslPq89/1f8hJfjixY2dQdW7Xk5m3t5ptQxWENpw79OMRJ4cDW4G7LrtFONom5KQ+9TBZJIhNgJobkqI4qwinrWF8qMW5 dcain@fullsteam", "key_options": null, "keyfile": "/home/dcain/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dcain", "validate_certs": true} 2021-02-12T17:50:49.801 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDs2+6XwRPdiDHT+63Yab00VBulg96/OLEORtNUvm7TDRMpcx4OD2oOnbsE4rG3dOdXNgT17H15c6MWFX2a2uC2h1j4hJTa+dG567HrYXRHG60Kwg5Sviiuq/ETPoH/r3OSvHs0HJ4Y9qYPQ5j0k7gG1XgIY3QHDWNRsXiyRXP9FEfoaOKABDTUoLPjqRX+4ExJd1Nz6DLxqIlre8wfqF5L1l12vec8JU3HB5NSRa87HxKFtLLIAtw7B9H/HggqX5wWVEnXq2qN4o6OOePhaO5cjWAZxOtT+AJTyHkeyaWLwIGOHBVZS5BKti6jLc+PxQxAflYlu9X3xh/2FlA+9H+ABZwMHE3ruUfTVbviYBbtccMCvkzcvFgnvaUCHjd7Iv/T8/JHNfBgFgigcX0Vei0+PZQ5oC5b7YpKT241NA8DCGZNgr1KuL9LazkvbkbpDcaHUOrkWsmSW1RRMC6ffYkbAI9QiBJFpImDAWP1u+1yCGhzPKBfGNt91GcLCD6ML90= sunny@devBox", "key_options": null, "keyfile": "/home/sunny/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "sunny", "validate_certs": true} 2021-02-12T17:50:50.019 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC4pIcNiZKVIfT9GxyTe/B8G2GyHKxrAle3iExEzLH5bBCKtBrottX1y66KBzQSsrAb0z9h+A+uxlswlY6VM/gNos9VRB6x4NRB4ADY1Bku+JsKs5aXwRB1zz2ueZhrFN9hJwMl1heLICx85ksHBdm0q+JPdUAV+/LBdr0H/IjWveV34WL3Awf7QikzyKhzUciMB2gN9PJXTW0bl8PP56y60ys++L3nMSRJ+15uPcWl8MS3D2vABII0Tu+3PsxzrTTNodElyQ6G/fwyFfWhdp7fkidjR7mDDx8NyWfCl2rKejXv9LnVC0wb3tFoQqvwdz6shN4E51EIEGXHcLUeRpBhL2pFOqIZ0qfnmLIQVZ88RKMTgo42CgtmS5UzxCLOyKOh9GxF7DDEajTinJFJJVbNJ5xF494H2geSnmP1o1AecGsGbCZ4ndhS9unSfIZLAyv1J3wZlAlMsvxXgY7yrh1srUiASVyKK8jba1IT+v3TXR8kQUCNsLKKbvuX/l0IXME= xuxuehan@bogon\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBd4eSizc2zAkWDy3mFZUhELO2wXDgkVxDNM/Nw2zT1li3L+6YOjFfdT4NOxpkg5laLd54bdKb5GVRA6iKqAO+aqo53xHHJGC8OY5eJaUPjtpypgIWO4Qzbos2ZzsC+nX5jvi8O9nM+JkabSW5iFv3gGPTqHixFYkxK2NIBa/UtXFOhmb1wZ6CG5Q+WObCosBUEMWokC8ntCADfhi6qDsO2fol6mYUhYEE/0nhQ5uLRrVIM5QM5dhrQVEeAs4KcYpmPDw6q6AKgx/f5zObh5OSJFTVsgx2cqzdQVJVDDx2OD2QVRtZ9fgvI2EGmKSwe1TUQOdqa5bmsVIp2Jfi8MC1 xuxuehan@localhost.localdomain\nssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCwIokXSvPYP3TRlSw0SBgmMFW5jVltGf4c6CqjIW4KKCgsL4c4Vs0FztRyDGc9p0krZUoGJCZHErVToXm6/lSq8JlfrlLX4X4yh37fUifWqc2UTJGeH8kLaP3WEt9vT8eQ9h8t2GqlMlGQU8KzVAdC+c5IwrbGRsS6S7D09/y0rklkhdJ6nuj8cssN/2TzLF9YIvVcFZHdyznk84eB7TgpPKD88MOpTnEgWFlD4vAWEuHpkuxgj1KI9LscgSFcW9/RxLeRxcV0GypZ8jPmczoCnin/ZOstCTpbcmjL 2021-02-12T17:50:50.021 INFO:teuthology.task.ansible.out:d+LW1kgGXXQSkf09iHeJ/cyCw6ooZOkynVwwmI9Uz2ZaMedCUBpG0bKclIerJ9jn+CoTs3U3Orhd9+6MVtCHX9VNJbVa4q0KCS1sp5S3HDrYkgUyawUxdYorrLZcqfRHWWo1luiridmuQdr8lQ6dvNpz4rhsCrAG6S1eeaC/s+o0Rzpv1YxU64JzquZ6WNR+W3U6Bg3iYkSoDCK9byf0rsptyL8= luqixi@luqixideMacBook-Air.local", "key_options": null, "keyfile": "/home/xuxuehan/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "xuxuehan", "validate_certs": true} 2021-02-12T17:50:50.226 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDMOuxxTZyxz/TEVA+ODgtLrNcSsUhKE64BZg/Ub96LLffQXQc9SYT/0YI4wxu2vSsky05PnVSON7jqGxunCt0+AdRqxXhW4ttmRIOz8rq8SiMnnLayV34nIF1G0aYTkgY4ej4+VmSVIHU3bb5mp/Q3D0hxQwDPdgOJWNyW9oYerHbnx6lVgchvsZHB9VO09GNHcAsyUz4p+xBf9iR7eS74p4dZmUbth/oBBqpOa4NgNr69iyZ9a/pNDLxaVQoIpDFrevk5UU5nUHQq2frFNxF/Y2Ja1EBXCavvPF/m5v3J3xNEu3SI/GbLFCPKCKeBDpXJbe1QqTBKJjMl9qGmKEJ3 aclewett@mac", "key_options": null, "keyfile": "/home/aclewett/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "aclewett", "validate_certs": true} 2021-02-12T17:50:50.432 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCdTLDprr3DXtsgiqrB/miglJwSkY7BAv+OW9sC0ZHQf8Y4dFoUtr7q9fd6W7ehKL6aMjya/8eQSMPRoEIZFD8f2KPCuE74Wgwgv2cpAkFmLjcPtjixqLo3LGFCyEJWGLQxjlthdzx59mpCWgBsEJQmlKP36JIYjILIMtXtsETIUGFa8QPIn9sMdIgutPPf3yB/6DHcR7C6yIAuvbJactEsiCt4C8+uJinvCii3PzYOMAEd+mictPGk8nOCQFZrI+ZmwTz4UbJylxFqnX8lhhoPIUEfNAXlb3lkZWH1LaFOxarbmupuPYcg2rT0+kFtyNr3/mAx0FO01jADYooqcW1Z cardno:000604936806", "key_options": null, "keyfile": "/home/cblum/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "cblum", "validate_certs": true} 2021-02-12T17:50:50.640 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDPUCIVfNQOOG+uAeKWo7tLAdgGO/oQu4exTE3XLfyTVdd8u6pbQCmIDFEnbGgMtlYuWJlCJtxJIzEZJ5HIwf+po3y7kb7HxEeWnEb9DsqJB4nCRE8fHnNLgF3bh9gvE7sOZdNXEKqtsnGcaaS81oXQ6QgkFGQzlOBOqxj7jIvad7AtNWD3WbJi/uTVDsWcukU51tn/aEW+hHMD7Y5QJkLH7O0PrCeEZCUCWoiMP7sNwACBym0DsqX+POkGJOaGx9EeNhGVA4B6o3xYXJPqmFbgMWMfG3/Y3SiheZjxzySMQ7FKW4Q817NaWqbjtLZQsSZw7JNqNFkcdTzVZwwCOJ1AAecajpniKeQLnyiWQ3Z/EkSGGwtDj67wo7uqT85tpOyLfYhQm6Zp+xSh+I6s0YJ0xO/huLKzeT5D1r5vVMk2VdcLD3CCxXUwZgaMUxYAC2Wq6mpIkxVZFQLbGuA/dWkQbFsrpbf7ES+kh+uzp4R6dQ29OjXbVcSfF2vyIuA+VUKoyj370YOZQhs50yuAJ4BSHlEgoJdRzi3XUX48h45XHiZcCY3X8LTX6a+0KAIgqeAWdFhRgQtAxoRYjhOxC3k/boBc8sEQ3o4pkxnhtE5Y/tnM2CPkPMX9HWYirNf9GyZTYcR/5rDf4ETQ18G68WLaNK758OuUPjSkwttcyLd4dw== mlatimer@suse.com", "key_options": null, "keyfile": "/home/mlatimer/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mlatimer", "validate_certs": true} 2021-02-12T17:50:50.847 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCYh28a9R3a7jpRfPqVAL9EcD+S6KkZzKTfCmRn4s8Fg39kUZsgnWUZQ3jUxKSeBA00te7WMR8CsIEzulm7f+/egNuPcvzNI/SaaCDtVMaxqlEuW85ll8uBQi3s4SLmOu9oS+5oid4cqTrGTcG6uhl29oWS/53SxoGh+T+y6GesO2cYRuTpcUHxGMvIgZe/bKtVGdP1y7rQd62e8emb9UupMBxaHpkYj9AaMuLjovHbBgecWbe4PlzPWNzl+eITVk2d6CxxIem4/bw9VHkRjQZlhL2nbwRgtczLYhlaZAd4ePEtsqA+7nTDjTDhXXHVIf1pDAsJbZEdgQCSGH3FzxtrQ2hZJrH/NIKdHk/OtI06e7POSGj7PcHzTvMjQSFnTpJXinxwzn0lvFSlmABN/00lIE0p2ZXfPs6dgWJ17JnTq6dMUoTiGhfKphPHyfSXJCzfa2VB+lQQ5eAaa9cRXy9pLmWGHXVGz2hsBMtyYmTjR/wr8kdTzeK+Jk4opBtGgcxDustg59b9x/jvgoc3HCdtnkqLTug2KmutB6kb+4KnVO07GsSp0RmNtDmCRcXs9YN/eia9ICtIGDWPc3YDRppsw1Vktza/+ZW5vGzIKILZTo4SNGcg19zQw4Qxd9eLkXfB66vE2CQvA90CGo7mcye+UWjjXLr7i0cDEszAloQ9aQ== srangana@redhat.com", "key_options": null, "keyfile": "/home/shyamsundarr/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shyamsundarr", "validate_certs": true} 2021-02-12T17:50:51.064 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDKRZyOnj6D9tCw5o4+ujpos5Uj1O0XVvea798DsXPRh8bEE1pLkARQJLv4iW6LsG3x/CXruahmwT+6C/VHppIsbgtpadMWdCTPYrsjctPK0/N9nr4kQEL6XeK3/GHFnwqYcc+LySpNNfbKrgN4qWOwdZBhQjSV7K7FJd4ZYSm4wp6TVbvKKp7o+JdeM2zhstfC9AcCtBv/TpxixFFq/+y8/Z0PwqN1YyHduDOr1dr1DWn4ZzrkhiTYfPOltx0jweD6zTJoTAJDqae7kBA+3XH3dUGX3n7Tchsa3bnIKVz9sLQvH/vPro61I+Xig6MQh77JsylNvpwb1LLOEzDf7dxl bigwhite@wrjj37deMacBook-Pro-2.local", "key_options": null, "keyfile": "/home/shuzhenyi/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "shuzhenyi", "validate_certs": true} 2021-02-12T17:50:51.270 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDEcI3X9mNxtHRSvi2gVFavu21kRCWDd1erXW3d/1zWnoUvRxe9ft2xA1c3Bmo067GqhFudPcXoYJvz3atAGR9k3n1UTtTJe95XRZ1Zmn/boi6mn7I2oRVFpW/KjxaiM3iN2SHKknMtcs6qRTioapYCU7u924oc0izqPBvkrC73cfVF4emH1ERWEcQBTxWub6N6Mn3xaDmJ5xnGlNvDY70CC9osoa2VRNJM7GU9eUQClHirPbSooDvvVKfTGD7YfMH76j6PdDR/a/cgNeiMevtMbeFGhu1SOEHb1ihCNoGq2vQFPPWwwTCzMI05XuUEc4iaxw/kbGBMpWbmIvswIMQrRwgwLZk67GGpf/dEeutI6WseeNKNuHnTzzx43AW9G2coGq0RKE7OuA5Cqj0CO7Kcf5UA4AqIIroLETVBrrEV6qzpkD74QBZXPl7gbSZIyxLpvtQoMfSSzN+Mj184LmQoURCqIT+yM5gddF5/S5I+rpw81vBj5M9ZskyzVQAo0H8= michelledipalma@mdipalma-mac", "key_options": null, "keyfile": "/home/mdipalma/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "mdipalma", "validate_certs": true} 2021-02-12T17:50:51.477 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC+tqgRGdlbb7NMefZVPCtbzVZBapxmmPcKaUtqT7E/9AKRoHuroZQ7KJQfO15POpIfEja7BugBt5F95aW2Bd9/xhulzC4/JjiZYc6ST5Na03BrcudI3YcktoGR8Mm7ZVzSojvm0SKtLLHM5Su17JpCC7BitMdao1mETlUOklEXc6pgbr9XHOayo1zeuos8DB90iA8TQ2soR/POD1Rix4GKniCHXz10+iXQFYPsif1HsQ3rh8Vi1M0cTq98bMSmfsKODil8KIxf+lvLCL4GAgDYEyxnstiTFh5GAcIMOOo2dl1fDb/Oa981CxWU+xxJD+M7AEJVeyAqqAnShvZU6y1L kiefer@kiefer-uchi", "key_options": null, "keyfile": "/home/kchang/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "kchang", "validate_certs": true} 2021-02-12T17:50:51.695 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDjj3bbbLvd4IHeFrSJyM9N6uKXA7ElQrzfcisKRGjUPON8IsAWkXQeAnHoBsIQv7tRGudXOrK+tbfv7crLI0/CxL+o0cCVW8bxuTEtdz07ZFsND9cppCc7Yq+wHsg7nQ/n0QfxnlYc0T9pATXUoT+aW4c3NQoDbsupLSkNs1C+G/MEBzz6iH6tgrv43OAOUccfQbC+IYx4RIKL5MteEjVrZanNELIG+jsYe+CZLdsONk1nR8+xKQw19Sm/jkKdl1K9NXoVPEBQmFXrYl6G+7tCdWeU3Vq24lM1X5p1mVspcG6FyyctcuP/8+zhL2L208DrXlOVPcqHR/4FEf3Wmk89 vereddy@vereddy.remote.csb", "key_options": null, "keyfile": "/home/vereddy/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "vereddy", "validate_certs": true} 2021-02-12T17:50:51.902 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+EkUbnaeC7Dqtq2N4/dX9v+WGlrgRS2ygHhlbZy4KHbd6+Q9T5NAWtqnFwqcbFpYYUBwMvz8zLo6HYN6gNskRN0qTiFKPQdtnydF0Tx1KHrdGGcX3G2lxyGH5P8nrqReXAPPvRiVc2/Mj7APCTmY2JftkSS1qYcTleeueivVyho6z6uD1AyC7OXANhtn7kGeIY/NvVKBhDr+3TuPmpPwPCQMcJUyZWCHYyp47v2vzNBy2Etp7WjGwOnfZ5ULiJGjH6PCXx0YfpmTUOPVkNbV6Om+0YJ3fcLlmQ/2nsdsWfOKvoAr+1uzO81h7ZBX9LVEitj8YU3WeSCV2vqS2VvOpq5NyHXWGq+ZkOCfhy4Z9MvMG+6hqp4p4z1JMpKsZ191kP7eAQua41GkTSQPlEfF+um1UV6ZjNzhB68utOXyGDoDcxct9oOdnhhjwu5W4/0t9iBHuMn02iHRYVrUWOH2FXVysR0p5x/L2xdvNPiTgFAZ2eN+EFWmdan4e9mAAw3M= gbenhano@localhost.localdomain", "key_options": null, "keyfile": "/home/benhanokh/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "benhanokh", "validate_certs": true} 2021-02-12T17:50:52.109 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDBYtdHKBXQcXQFkEW8fSwlFK/xZM16VcFY288ABI/DPfejTStHhxiRvmlsWzMy0Ar5WadZ8FU1ptjA3MKBEY1Do1r1Ul369k3zHCvxXyg9QGGMNPYfHmnNeoF4GEgoJpuounPs0mkLmrPTR+cR+/XuaZZt38nsad8M/zJetnolvqCWpcV87D7+Rt8F32J6kGDknC4ikEe8I0n7F+goOmJhlgkqegbKEeJ/EMnOItwVZHFwaC+sxOq4IlYp2f1aZMKGAtr5+ICwYH1SMCcvnp3qWL8O//t2tvguZgQxzIs6Nf5GaxwdQkh4CvjbWiCkIbGFJ57YICS6k++KMXMbEnwh jsalomon@Josh-laptop.tlv.redhat.com", "key_options": null, "keyfile": "/home/jsalomon/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jsalomon", "validate_certs": true} 2021-02-12T17:50:52.325 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQD9ltUNitnn0FNwMeiGkBnozpBN5ziVR2MreaqkfqTQY2TgI24559xuILyy+TnAKumhemZ8i8nIQ876Iso85OzIXbUbRM45hpL/bMQX7pWyMIRNeFKrCDwf18OT0DPc8aeDOAFQ6YoN0j4awUENjdZuuAW80cfcjoQO3m7ORF8wBF0IvMWwPDIQQrz+Li31ECgXg6VC7K3e+BNHlfNQ/2/ahS0+V4J6JYFsylZL0r2tydhDMd6rR8QA9c9BgeBz7+KWobGtiZVe7KReTzJ0O38PxNd8br3aS50j5BiN63gch8eFxPAROaUMnuiavu6OTDy4/aeL3Zc5WpUy3VyyumWf jolmomar@localhost.localdomain", "key_options": null, "keyfile": "/home/jolmomar/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "jolmomar", "validate_certs": true} 2021-02-12T17:50:52.533 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDQKiUAlB8dxudAY60M7jnIJvs9yb2pNaT+BVv1Vk1kMs0XjPjcw9cyT3DtQ1xGLF367phRi/cfQbXOYbHG+uyBzQ4EEnBFvziktWRx4mfeWYxKjkjbfkVfpKPsxztqCbwm6+NYzMxGK2cExN7jUHnghHAjfA0IUpVVg9peBD3kUveLwS+UxuJAnuj+G4480LS6GDyLrhFEXU7uOIIlW4S9oLnU6vJL6GAzwyLE4ijTA8Ek03SJ01Ey/gpU9AqnQNp9u12l/7sJely6sI9pAuIzmS8Ht324hexxSWN5sFjXgLvLfFK83ziDBdrFBr0WXnWsFdRpHs2CwSq1JTkByqbN hmunjulu@hmunjulu.remote.csb", "key_options": null, "keyfile": "/home/hmunjulu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "hmunjulu", "validate_certs": true} 2021-02-12T17:50:52.739 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDoefxOfeArToGucfWG7y1tyLF2I1J/IkBSlmxxEJyHyDUB6ElXjYIJReoiPfmB0xLEmINpmt8QA4XN/8UbR1r9LFOexlZZbX1LSbihi11LXCvfIJWqkHVwIyKHRTu/duMrntBFVENJ0LVf0GSWVTtllXaDA+fMkp6/ra/hlRHQbx19UXy/9+Pogj8Xl+SntnMGiHlX8FFq2A2tUqhdQSf/1GW1HfUIa5WtwdEqIpKpAgMM7sOW9i3QAmLAMo8HiKrArSiy7jrRUODCqsTJsfifxEbmnOS0ZtmDh0eeWL/Dyqbz99Rg4WbFNeK8oyZYr7D6bEAhd2az2A+wycPx4Nq7 julpark@julpark.remote.csb", "key_options": null, "keyfile": "/home/julpark/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "julpark", "validate_certs": true} 2021-02-12T17:50:52.946 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQD03otezqmbR7v9dq62mWH5Kkg692jdmDZk6Ve4vux0A0YhxBuXDCyBWLE1Z4lLNxQahvK0pGsgokp+Uk+Q6aU+ZnF/E2jLp6YWerFXIHY3M7byObJx9+QuVBfNI0bZBkV3+vSjhSCGOYU+TrTHsKZqhsTcejZjmL3gFLmWfU6dTh0sX05qz97cdJl3CMPhCIR3KjtkiPygj417/N7wet/Efv1t79TvyS7dtK1oz+zRLEpKi/fZjvpXb8xec7as72Y0XexsWMDyZJ/N8PijTQ5L6EfLMV7ae/bWGzNAKsoC1dHVY7q/Q4pzktC6zU0Rc9sDRMtrgFg+q+04BfveSLfxTb9tqf7SbSvh7IAG1eKiEuDT/uy3T6Uiw+CiZ0eIqYzZscO/aDjUArGbS8eyaycybdYCBK9Jz38i0mYKkXg855yoVuIExr6MXieYwyVoVW8gbx1bAslaafrzra2dkAFByOJ0sbosPFRgR2+boUSxD6IaE+ALgno4T/0CI2RMiH0= prashant@cephdev", "key_options": null, "keyfile": "/home/pdhange/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "pdhange", "validate_certs": true} 2021-02-12T17:50:53.164 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDg7xW5+Mk7pG5MjLxOc7GTvDTCPBJj2L3MEQZIapmmo6t8uPODwDewXncSzrKDH6zFVE4TTTD9fJdu+BhyT+jCFgXrEOMMabj7vJsg0wO0SWqr+4NECFPhD9y4ewfYiewBeXisclA5Bto91v93AtYpZBCdmPYSi0BEcPVc6S3iBUD0PzIrL62I/kBAtm0f/bScRiT7XvQJXvkbEObW2yS2XhUv2qgq1VlFvO1jl3B85KMTHVb8+LrKpQWOKfRfxJ74c51y6tnEjZLFO6Xn0JevqOhd8gPQIXueFZOM0g0pNp1mbtGaaBlEIibbrzo2p9kzsdvJeHMq1XuMIi1/Qyj9 dpivonka@dhcp-41-144.bos.redhat.com", "key_options": null, "keyfile": "/home/dpivonka/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "dpivonka", "validate_certs": true} 2021-02-12T17:50:53.370 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC0m+8wna4RnrOcuCUj971EBe4cFiTXVkEWkIsUX3yeCUwMaS/ZjaoOjcdYX7cYpCp1ilo4FSO5BsochU0RiMbQEwLGcE0zB2q1ac8+yYKCuWWyknig2t4bOReu5OrXvl6mj/9R+p7clFlYhwIoNG5ZZ/2eTbo3nH7gWgaOKaPbTtvEGg/5R/qjQKg8OTC8oix6kLsaNTisQJ78Zd7b4ukg9bO0jlq/2UhQOBM072sU1/kJpWTkcA0xxm3nsC4rWPwapgz7UwBmMdyx116GZgWdfQecTLKO7b8pyJw21npk2ntUmeMi7WPZGX6VI3IpaAhBjW3KiATAb8JbAQh8F9bmejkw6x7zrP3hYKQOg3A+U8DKRRo/Ad+f0zz9gjC42DUvMEOr09RBdxZcEAPY7H/YRWEqbA2zPX3Gfth4b7/letzM3m9ToUvVfSt1CdjaZD+gIia9pCgDMb+j1j2w9reG4F7Us+W/1f5pYCyFyrm4XO3AdoYM9/FOAOR3vffTIQU= adking@localhost.localdomain", "key_options": null, "keyfile": "/home/adking/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "adking", "validate_certs": true} 2021-02-12T17:50:53.587 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQCs6q2b5NcUsNXl+w1Mrc7idNvGdwWkL0XRX5zTw22UNpQyqPkSrRTFKi4NxoY8ucv8JoLnaJL7hYSRHKTh7SqH78uPLZX0FLtoHvpYLgUgJzzJ5AOW5uekIhCMCm+3egmFnEBwFcNxtGnU2es02ByXNEXRxt4/LQ0f8K6lMqxVRSKYcltvWzlA7WFFdTnYMunbCYOdpw88RXm2pYjGIHRBnie7+pFiFSS6K/mkzQcfLiJmahgLSaYslI0ho1vQd8xfNLmZ700C5F4VTUlT/mJK/dI2LF3OfaPTsyqKNzOchrjLFx5b6dhZ4WZyqu7VKsHNm+xHZwT8VP3LPGSVVBdmEsiCsmV4g4QTI5ssZLi7siZDpaFp72nVAIO1i6kBGuiF3or+J4LmIh8ZRKzTc0DRDOw7S+IkWu1fv2UhYg83z6wrz4HeczI+bnolFFWW6HgGV3MFaUeRTRvrbxw2DPoEGQ+cjP0wsjyEhxlScwdh8W8IbVpNl+7mBe45YGUH2hg83Lst5QQxD4PHelriu40moWRHFKCnKfNesb4L1uqu29o8f+/PgpLdrWE/P5zIKe2ReWJyScYAI7jTMlFw+xgf+0TKryk44NMy90l3ei0umlx4qsFbbEv+lhzAALrUJ/da7fsQw7eIVykmcscBeYr99/YSYk4MGdk4jVCiHsQkdw== amathuri@redhat.com", "key_options": null, "keyfile": "/home/amathuri/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amathuri", "validate_certs": true} 2021-02-12T17:50:53.794 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": true, "comment": null, "exclusive": false, "follow": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDr6H7hmtGCUrO3GppgMHwbmA4XT/Vipz/uLG/N8h6ZxtaLQKpCefRpdKcFlOdDwxkTmT5q9vmXQ8nl95lTGq+eeMxl4ho104ZD9e9AIYmgYqFy7/qNP8gkqduIPlypLL9m5rElTcGd7zqfMqgljMvAmnXLtZNJlIuFtszVU5WPI1QCwfQ3t5aHF6j66bpVoN9uWItVGB1LApQz9TtViWsQ9Hl8X4YauJToO+TkuSX+H/OGH7ZC6CPOzNZb3b3vF/jLMnTzXl86PJs1se4DjKkRGLVCVbr2WhLwQeZ8TEn3Hg1Gn5X8zNM5cTo6U8SnXBAe9ZYbuyfvghrNLdlZpmN+8l0/l5zC3z+N+sEzklXwHw4MQLF+KSloVFd9NN/kyenufTZAyp8VxllZ9TgdC+CPORIWyzmu1elOtYo7LNuE6CIh/Zm9B54SO6hVHhVoOCqe/dIyjeHJA0WKIZblK1XZEvWuu0E2T3jhxfe1GICyTukeB8WoBQH9DZ1yzMPt59E= nia@redhat.com", "key_options": null, "keyfile": "/home/nizamial09/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "nizamial09", "validate_certs": true} 2021-02-12T17:50:53.884 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:50:53 +0000 (0:00:28.609) 0:01:07.276 ******* 2021-02-12T17:50:53.921 INFO:teuthology.task.ansible.out: TASK [users : Update authorized_keys for each user with literal keys] ********** 2021-02-12T17:50:53.922 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'andrewschoen', 'ovpn': 'andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "andrewschoen", "ovpn": "andrewschoen@home hYDlX2i7X8t0JHpK2smvFA 2a6e753fb3c401ad75e1273ae5e18e4abc0404149dbdc3675bb317a63a4469e9"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:53.932 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'zack', 'ovpn': 'zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zack", "ovpn": "zack@ceph0 1CeZnO8st4CuhLpH/DJNVw 7230fda728e34998da4401284fc3e5c8939bfdd52482f8cb675d8acf6fcd12b4\nzack@mba.local mzf7/v4tv3uVExt16N/w/Q 13dcf7082b974315dfab1c76885dfc094d7ccaf01ae2e7898a20005602be2caa\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:53.943 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kdreyer', 'ovpn': 'kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kdreyer", "ovpn": "kdreyer@redhat.com yUBHWOGnPts8xuDSon5ODg 38feb4fec9f1ba3de74ed074d745ed46a8e5b3ac1d44e50005a04cf07f0462cd"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:53.944 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dmick', 'ovpn': 'dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dmick", "ovpn": "dmick@angus z8fyazJ0pc56I/XTlHAOrQ db682b1b89c790bf26a2e12dbea03f4495f8ea5f248e29b2f5c5dc072acb3c93\ndmick@home o9fblAdxhBToEJxbutv2mw 1704b16322c3984ed565b48cce52877eadade26b4093443b349b70f846f0eac6\ndmick@dell r6meTomZVVH/WVOmVf2aSQ 8ac1a9d0452166af59894c6b57ed07a12b42042aa6d2618df2bec9c962684072\ndmick@dmick-MacBookAir EFN6/5V1I5YNm72XjlApUg f2335a3fa49a0296233ab5faba28e7bbf309a89c193169dbe3c411e4219827a4\ndmick@hplaptop 49Wf5LFRBuobkfWp7OoAdQ 1401efaf459e0b7673f03b546eeff2c8144219c674eb3c94aa23e5af5ffc66b9\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:53.954 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sage', 'ovpn': 'sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sage", "ovpn": "sage@autriche 2Uhfc/1jXje4W7G4s0sJEQ 09631754d059fab96e16be441b3b8f7e63b37a260677897a2669cf7a2501d2d3\nsage@gnit YuLp2iihT6cCqY+GmW9IVQ dc79b19c45b9be1b4e69cd7debeb5b87aaa8cd0050353f798913da22c233875a\nsage@maetl wpsQIdtAb+z+ietqr+90Zw 42c518015b7cd440e7c8afd65fcca5769e31e9ef8f8a67d109a3de38187087e6\nsage@lowride tak6ngfj0ohW6upHRUMq7g 450919c617f452e874ef2a8cfa5ca18f47d249450ef0b0374849e721b0c53a70\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:53.966 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dgalloway', 'ovpn': 'dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dgalloway", "ovpn": "dgallowa@laptop POebac0SJIVkSCyi0LE4yg 52fc0cb34f3332a5f3cff117a20f00e8ea89a051736cb543a689a96dd9528ae5\ndgalloway@p50 7PHdbcNE/YjcHOm5NbNfpQ 7075211c35997b01797c2f15083aedf13888fc4c54080130342dccc76469fb40\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:53.966 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'akraitma', 'ovpn': 'akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akraitma", "ovpn": "akraitma@aklap 2zvxeeQ+bQ4UhKBE9Kz2DQ ff4a9dbab841ee62e7328d8cd0d285c68c22a1de6e23ea24f470b3dcb1a27045"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:53.977 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcaratzas', 'ovpn': 'jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcaratzas", "ovpn": "jcaratza@fedora rT6gfdsBHVr0RvokJfO3mQ e634e7a31d873f310ff5326bacae6c19f95087f8215edda618f68dd2761257a9"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:53.978 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'trociny', 'ovpn': 'mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "trociny", "ovpn": "mgolub@mirantis.com +WMPxNCgiMwhNJAJHNCV2g 4cd6749ac740ea6dfb50d4f602920ac75e979286439bd510540a8a551728104e"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.206 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'smithfarm', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf', 'ovpn': 'smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "name": "smithfarm", "ovpn": "smithfarm@wilbur vEk7281ao7DkpXE2Cn3rpw 20058c04c4991caf38b9517faedb75c8562e8d8669808b6be9f70da6a5b64007"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQC6jqQtqO7pnW39mpSQQC4L2FuxPOx/5j34lp2PrwuTWJoDwtUVCZhpNwSs5UAj3wdXGXCzWYWxu9d9Gh0UEBkiC+2ZHtLvWrwS1TDVRy5g5jYVm9vX3kL5dhoC2vYPyQoQkxeMX3BYlhLwkSjCAnjaceJJZyOT7KsqbpanvJxvudMzeyCosntnMuxyPzjH9CkVKqycddZnscdHMkDIZ3TjNhnL8GKK/QCPB60tkCO6HWN/g4HxidRjrp2VjQgIa5MWN9LBR6mGspZI6+rR1ygLhbl1scr1cCCumh65r+zmCC/ZY33YCfRXdmQZo+ShsBa+KbcvakEx0Bqgx7DsMoiR ncutler@pantograf", "key_options": null, "keyfile": "/home/smithfarm/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "smithfarm", "validate_certs": true} 2021-02-12T17:50:54.217 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mbenjamin', 'ovpn': 'mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mbenjamin", "ovpn": "mbenjamin@ferret N7Ln6KHUAYT/iiP4Swkb0A 244b90574d7e294f6f80949ef5e640bab103c2c34e1902a69cdcf592607d950c\nmbenjamin@duel PRRFsQiomHSJX8sVgRHblA f3009adb7502a6c5554e1157d36c63d84f40fe5074a01d7d0280e5c48da374dc\nmbenjamin@rael 8ixfNNx3j2m2+nGhXkHsnQ 62fda7ec724f0fdb3f05add4c3739e07a5c7eeb2380d5f1f7eeeb13db35e16a8\nmbenjamin@shatner hJ1oEl5wSZaKPJC1Ugss8g 9c9161b53da4d842308f80e5b490a0c1539076f129bd0e34ee2253272dd55c21\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.228 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'aemerson', 'ovpn': 'aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aemerson", "ovpn": "aemerson@seasalt enqtMOCQ0a24hizRK46SZg 42f36c9c14c1eb7c468e41cf4c5649e30037f0aaf5eefbd6c07be9637224ca01\naemerson@electric TjQxreDKF6bsIehyQnX4aQ a73b7d56b95ba3d4dda5ded995b9cb98eaaf5d2db7d148c340aa4f4c68cf26d7\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.467 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'amaredia', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour', 'ovpn': 'ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "name": "amaredia", "ovpn": "ali@freerunner yQjRpRVG7D5KN2HAUjI30g 9d677a34ae98477e6cc8ba1d975d81dcae43a102013b265c63f3ea91e7dacd78"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQCc1ZbHNXuJn7gxyt7QJ579gEM13MuFdTzsrnedYuRIW2Zlm4rCFr8Oj5SGs5DWBIJnd3W4O2v7PjuVQMxU/zbwnj7mdwBmLFe1cSzOJv2eP1R2uaU5z5C7KNmyPLU++pGKClzb6v5wcEQUq2K35xnuXUU9B935dK+Fm7bK7+HAxj+1vpVeycbPFyPhf6mwbx8dZv4uvZGV2+CGBuyIB/5U2AMJZy9LWim3AR35bip4ftXvSKlAON+RHhnS0toG/6uwp0XlFuGn5H8snaca7L6hGtB4xg1PqA5aMf33Jiv2NVLQo8emHU9J/HeNVS7ksoSZ6InynpLZ6b9uXa9OM9XL ali@parkour", "key_options": null, "keyfile": "/home/amaredia/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "amaredia", "validate_certs": true} 2021-02-12T17:50:54.664 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'name': 'tserlin', 'key': 'ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet', 'ovpn': 'tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5'}) => {"ansible_loop_var": "item", "changed": false, "comment": null, "exclusive": false, "follow": false, "item": {"key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "name": "tserlin", "ovpn": "tserlin@annarbor DlKe+OWBPcFAQtWMUAHnwg 6b268bd737ffa5dd38865575ccd444b92cb912c70f5b82dac41f9c50505df4a5"}, "key": "ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2ok6CBUpOVGv2RFws44GtEP5SVxOi4Vie0WSZYLpD55rTfmOtsItN1d1EciVNTUyWuyzMeQGWC4JAd3/2l3gR/5ZwSvd7b/7TCNYxjAMEubRand0GxEoiKhpkJMMmJqcT0KefP8pr31MASWPuERj1+0/IbjJExsvrJaUjqeIfZ+DWR8dC2VYdcH3hsp6AE3mqKX/9693sxe8ROt6qY4WkpZcO4M90unOVa2CnJsYqKaaIC4z3fmKuHZpJZjiJMrg8rtuN4r7bnKWPEVGcahj+i74JWwKR5+2gntLpxw2chIBmf4qFu6HDplddig4V3I/2NLB8soBpgc+m8O7YyYl0w== thomas@easystreet", "key_options": null, "keyfile": "/home/tserlin/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "tserlin", "validate_certs": true} 2021-02-12T17:50:54.675 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dis', 'ovpn': 'dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dis", "ovpn": "dis@zambezi wXYUFvWPBlkCFz+mC4RD6A 11c464dfb2a27986e029f1915732a4f237baba4eade02bb045c8f0d13dfada28"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.686 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'gregf', 'ovpn': 'gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gregf", "ovpn": "gregf@kai YhNrPfedhZjbvGjOfmotOA 440cf8595a87cd307790bbf79c3668455a0405945e2b271d873325de222cd72f\ngregf@pudgy VZrk8nWzg7pYLOrZru8dBg c1d1a0e469a134ccf5f5a5525631a6b83efa6970beec3b23809eb0daa3dca47f\ngfarnum@WorkMini2 +bAqcTdU7Ok9bGMcB3A84w 3cff1326561a23cf81dd6495373cb83ed149cee026c6374d72c19b483f4f1f07\ngfarnum@Macbook bxVtolCC9SY3QNlpx3cE1w aff8d28bfb4d693253511d29e8d399196e964fc096594ec705748a5469d44654\ngregf@fedoragreg Jdn8I/sFGcr5Aa/dici6lw 50f88afc35c05ef8454742226f7baf2cd20cb1e2d4d0c9f4a393013877736bfa\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.697 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'joshd', 'ovpn': 'jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joshd", "ovpn": "jdurgin@glarthnir ElGaAgbs5VZujuzsQMfCmA 2a156febba037d02d1099bc11d1e697d34300b2c420f2df664b5b0de1248f983\njdurgin@new-angeles jqa015PRJcHSp5WHcwJjUg 42113e1156382fde866d691f30584f6b30c3dfc21317ae89b4267efb177d982c\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.698 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'davidz', 'ovpn': 'dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "davidz", "ovpn": "dzafman@ubuntu-laptop-vm NY9y9tqLY1beEcXDwMavsQ c869d42fae1890574577a8014d97d1247f1a13cb6337037d2714f1d236fc65d2\ndzafman@ubuntu16 2a0rAy5QmNFSEcATNz2h9A b7c11fbb0911fc4ac0216a1a8eac8359a9e8f43d69126db6b45cbeabd358c2b4\ndzafman@ubuntu-1804 PN1pkeGHGloB0K+IZrfB0g f1c01b447b9ec3fc048c32f606a33fb488ff621e11aa305ac979501030202658\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.708 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'cmeno', 'ovpn': 'cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cmeno", "ovpn": "cmeno@boxtop KD0XJIGun8xcnNI83pqrsg 79079fe78e1cf401cccb14692c9d16f63e2010c57cfbf56c487387affca90aed"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.719 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ivancich', 'ovpn': 'ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ivancich", "ovpn": "ivancich@ann.arbor Kt2vxZ3Ge609mHfjx0W4Cw aaa55a9e2b5276b62a21cd3c401b365c5c2693e39efccb2f9edefafefa1dc8b1"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.720 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'wusui', 'ovpn': 'wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "wusui", "ovpn": "wusui@ubuntu /nElLTGqxiOr/hD6eziF5A 2e244e2b62fa42dadf3a3a1dbcc29475410cee6550b0c6b3603c135853937551\nwusui@thinkpad tu2DxDcllIdwb5ewldgT0g 1590a7d9f1377b0094e9ba4277e7bcbe6374791f0b3d3df93026345c058c93f5\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.730 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuriw', 'ovpn': 'yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuriw", "ovpn": "yuriweinstein@Yuris-MacBook-Pro.local wKA8mCcsdsk/CE+1d9GiPw caaf5bf5eb31ef269e3d0bc34d45dd761c0bb0907172eea6e754434de4603db7\nyuriw@home 02TZyR3JHJMxEQob80ICNA 85b4aa0f69f6d469dae0bb3dca4baaf222e164927ed7eed2082caae8f4717e48\nyuriweinstein@xenon1 C9eVdLb/i18lMcMG20rGPw eaddd0e831a77de3f35cb19e307bd6f9aeb0cc03eff1ae58490d7db33ced4311\nyuriw@yuriw-RH2 WhMAZ+zZ/PPXOc+4HEDaZg 2d3e28b31edc3f2be43f6b73150dedc9c9df6e316d29afc1416d6b96a982e217\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.741 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'tamil', 'ovpn': 'tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tamil", "ovpn": "tmuthamizhan@mac /1CBLC6rxqzJzPspffZV0g 4d1dfff2e097a7fc2a83ea73eccad2f0e453a6338e18c87b4d89bf370733e29c\ntamil@tamil-VirtualBox M22QdmhkSPj9aEcTiuIVfg 8e76e06b14de99318441c75a96e635a92f5bddc54a40b45276191f6829c6b239\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.752 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jowilkin', 'ovpn': 'jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jowilkin", "ovpn": "jowilkin@jowilkin 2r8xOv/eCTcHr+HSsMkPYg 0ac416d2dc139628144dfa0822af8cc0a455f5f5f3e4d1b9713c14115c062218\njohn@osd-host 7zjDTxAYhCmTX+Az4SJaoA 7d924233fdef168e2c5c01258aa349de108629ef2ff90d17c0b96acf22dac7c2\njohn@admin-host 7cpk7iJ1Hg2vk4bPDovKmA 05765178f27af6dc4e43e47f52d773aac3bc1b3f1dd998bdbf479b951bfd2efb\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.752 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'bhubbard', 'ovpn': 'brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bhubbard", "ovpn": "brad@dhcp-1-165.bne.redhat.com 4oShQI9+vYtX5gA47np/Sw 3fc7df5afa772752d8eee15c01d550cc1dcc88b6e940abc9f9f8f26102d239d4"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.763 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'yehudasa', 'ovpn': 'yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yehudasa", "ovpn": "yehudasa@yehuda-2 NGXHeO0lAvRFfYLalffzEw 0d3489b09f48ad5fa9d1ec5944d9c8020daf852488f389f3d44fe63d3f651f34\nyehuda@yehuda-940X3G shisK5LjI6fr3ZBJy/xX8A 49522899bd26130086ce668079f0062d987d85dfa5767dd5c34e5953db97997a\nyehudasa@yehudasa-desktop OT1MhoO0WihhvkKztqW0Uw 12a4d6b54390b9df7f5af3bd6b533f3c1fee0c7b9fbb79f0a87bcb28b182c7d4\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.774 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dang', 'ovpn': 'dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dang", "ovpn": "dang@sidious w0CNW2g9K1WiRenVGYWNUA 4f59d761bfab3659115da2b3b80a486266f77b09d8527983217d15648b4f92b4"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.774 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'branto', 'ovpn': 'branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "branto", "ovpn": "branto@work ganye+HpG3dkMEik6WtTng 018f3f9b9d49dcefa701ea304a8e58f002c46f0650edae220a0a7ab1bce36aeb"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.785 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiaoxichen', 'ovpn': 'xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiaoxichen", "ovpn": "xiaoxichen@ebay RvJJ7BhIehpoPtggrwnskQ 862ecfe7e15dfab86d61df86856bfe06cbb99f240f6f03851f7f9e1a255327d6"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.796 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ffilz', 'ovpn': 'ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ffilz", "ovpn": "ffilz@redhat.com 6YdvqxkKfmDWGD2s0wA7Ww 4ce64d08686e34e559ccec2498df433b155b70c9ebccaec616b6b34f0f0c246e"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.797 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'joao', 'ovpn': 'joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "joao", "ovpn": "joao@magrathea eSS2/gvK7ALE6L9bITfuMA c3caaeecee3f43e39b7a81fad50e0d874359c70a9c41b77c661511c71f733909\njoao@timesink 9S3oER36HheVupjRpnLz6A 9dbc964184244e9da269942dc73ec9ebba6594bcccfdc0eb09562b58b4542162\njoao@tagide uUDVmZ1S5lp/Hx8eKTfDCw ee37985d0a51360f08266e0de208ea85616777a325d498d7c895a80d4723f844\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.808 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'nhm', 'ovpn': 'nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nhm", "ovpn": "nh@latte JCH9icAtDPj951rgbPKJyw 9a3d30ec1ec9467ccdc8bdbbfacffd7396fb38e046199ae37b2b7b69dbf37480\nnhm@espresso +YYZPT29wYzY5ooaRzabCQ 1ee041dd58b9ec6eb678c47632ece7cf6c24e23bcbac28a77a82af05ba6cc148\nnhm@mocha HgOGOfkBEzJihFsKmPRfKQ 2e17f3ba0b90df7a36f19a7c8f64d2aa8f966b2794c94caa110d313e927a1c1b\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.818 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'nwatkins', 'ovpn': 'nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nwatkins", "ovpn": "nwatkins@daq unHfXkUpAHOnzeptznuLLA 33b2003c30d0cc7a6b194e76be92c7b5d270c2d2a4b4a8b6e673f0f0dc1db313"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.829 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkidd', 'ovpn': 'linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkidd", "ovpn": "linuxkidd@zenbook oYp1WWV0JwpikHCWCV52Lg 9aca455b601bf3a365d31068154150ac63dd76f32cef29a55f9685dd1a88aa22"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.830 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlopez', 'ovpn': 'JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlopez", "ovpn": "JCLO@oh-mbp-jcl vZhmBh/1LjLFEu+atRec6w 1f13f591373b4dc798a9b701fabf1eb99bf4aa58f87b6420d6c916716f0965af"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.840 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'haomaiwang', 'ovpn': 'yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "haomaiwang", "ovpn": "yuyuyu101@desktop XAUS/1Geh1T2WY//5mRahw fda03bdaf79c2f39ac3ba6cd9c3a1cb2e66b842a921169f20a00481a4cd3d9cb"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.851 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jdillaman', 'ovpn': 'jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jdillaman", "ovpn": "jdillaman@ceph1 kQ96pIpKTocwIj9H1Udb1g 6c087451af94ac18c144712bcc9329799d86f6d90376839dcd6fa41cd73e3608"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.852 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchai', 'ovpn': 'kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchai", "ovpn": "kefu@gen8 HVoNrB5C8+bYxuqIfByeEQ 4dddde1890af2d6df367d3d832cc3b9b660160a1db69f0135e0d09364b2cb9b3"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.862 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@vumrao VaMHdnIGTl6y9LIkurxfjQ 71de53c4a0f212b8f919437d7d433d24a33d7a33bc6fe5df5d047e24499994b2"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.873 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dfuller', 'ovpn': 'dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dfuller", "ovpn": "dfuller@laptop 6U0lNVidRy7Ye/5q6cGN1A aea3d019f68a95094c99385aff099818224455a829615cfc774587e4519398a7\ndfuller@rex001 6Z8bfQDgPXVSGMeeQHoItg 3abd41920b72683fbba7f25be88ff992fcd753119c4d2086c12daaf20798e684\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.886 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'owasserm', 'ovpn': 'owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "owasserm", "ovpn": "owasserm@orit-desktop zi8oe8INwK6IZrtUz6lg0w b5f4f96232a77a260f7a16244af873df2ff143c2bde0dbaabf02e9952b8f38ca\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.887 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vasu', 'ovpn': 'vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vasu", "ovpn": "vasu@ceph1 +1D1qNUAk8h7OF9LF9qkrQ 963aa256fb99bc4b77e7085f57cf910a04c7f143603f81252331411eb8e37ec3\nvakulkar@vakulkar.sjc.csb O8ac1k0Dh3xkIFX8NFyIZw 471538eeb22384b58921e4f11af272c00c0a953dc7fe8d95ba057e65d141fbd2\nvasu@vasuSrv waJqYAARY/LnfuP1x/KQzQ 68915d3a1eb3dd00a562c149791cec5f43a96f5fd0b851ec855ec3f5dab496b4\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.898 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'smohan', 'ovpn': 'shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smohan", "ovpn": "shmohan@laptop 7wwHLZZNa4ShUV1imXDDjw 0aca19a8ff6dbeee2821dd75a329a05b8052170204b2b242ced9b1a68ca8df37"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.908 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'abhi', 'ovpn': 'abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "abhi", "ovpn": "abhi@trusty SKarQTpBigBobP9sLjdLiw 868a74ed21b46f7f64255897d824f4e3eb21f8dde844bbdaa386681c942d8114"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.909 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'cbodley', 'ovpn': 'cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cbodley", "ovpn": "cbodley@redhat.com W4gFS4oU8PtUOrdHuBYwXQ af7dafd992687f5d85a79866838a78e5070a4934fb0a935e8094adb31ec28611"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.920 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mwatts', 'ovpn': 'mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mwatts", "ovpn": "mdw@degu IPiZqcFT2BLuf2h3+tw58g 7af390a631ec11bddd7d1ac506d29af65e1e01e19f7dc931b4f459030cb7a195"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.931 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "pritha@dhcp35-190.lab.eng.blr.redhat.com GCk3PIYzUNZ6/xrsKoq8VQ acbfc4279970b44c3d008990e0cf1bb5eb280299218441a0f25fda988bc555f6"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.931 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdonnell', 'ovpn': 'pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdonnell", "ovpn": "pdonnell@redhat.com Q9j56CPPXipXScmKi57PlQ fb616603b6d27cf65bfa1da83fc0ca39399861ad1c02bfed37ce9be17cdfa8ea"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.942 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jlayton', 'ovpn': 'jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jlayton", "ovpn": "jlayton@redhat aNfzMdXOhhmWRb25hwXJIg f51fee42c5268f7b8e00d57092dc522b0a07b31154ea52cf542da9cac5885868"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.953 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rzarzynski', 'ovpn': 'rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rzarzynski", "ovpn": "rzarzyns@redhat.com h4UkibgFG40ygfpfKTMBrg e20ca28c60144dbabc97953cd4c273c1b92cd45ebcddd0f0299679d7a5c87d7f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.953 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rdias', 'ovpn': 'rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rdias", "ovpn": "rdias@rdias-suse-laptop 0bh54sdB69mh95n5rWME5g 452e3338e48d04d4e816f4f1cb54d637746a7acc1ffe5e8ed4c1506c8e07a72e"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.964 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vshankar', 'ovpn': 'vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vshankar", "ovpn": "vshankar@h3ckers-pride eYOkK3Yj10T5PFLIUu5DKw f6d9d714fb6a97e51a4c8d2a4fb7ed0c0e87caedfd0fa813b209846abd9a8a20"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.976 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'akupczyk', 'ovpn': 'adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "akupczyk", "ovpn": "adam@TP50 C0YuBT9bYaNhdDmjbF56xg 5d298b33b9dbaef364b037561aa5c5de374405bb8afead5280db5b212506ea58"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.977 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'nojha', 'ovpn': 'nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nojha", "ovpn": "nojha@localhost YQTw/h6ZMgdJn7EPBmbEnw 253574eae62759c4c5d3bc4bf949c59948a0488e4dfe4c91ee754a3b5494847e\nnojha@nojha-mac aucFu11aJL4iZg9ch76Lmg 6f14a860471fabc125d167c7af8d4729c26bc93bfab5e73d5457ebf54d32d526\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:54.989 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ifed01', 'ovpn': 'ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ifed01", "ovpn": "ifed01@LocalSUSE1 6g6hX1bzTGBTCnDevAn0+w f18c9354f6de3f371c3b51521b62375e474802ac21adb3d71e09d8d5bf9d0c43"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.000 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'myoungwon', 'ovpn': 'myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "myoungwon", "ovpn": "myoungwon@fg2server p5CiGtdZolEwS1dCiRJjkg cd2491f835be1474a7802db607cf9a31c59e1031001e1dfb8fe240616dda9a71"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.001 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'gabrioux', 'ovpn': 'gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gabrioux", "ovpn": "gabrioux@elisheba 80kx1htp39RsFrlGONcp+A a95579ef6f90694cd6fd390302adf8532237a8ea65bd5544d9b561654d712ba2"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.012 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'leseb', 'ovpn': 'leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "leseb", "ovpn": "leseb@mbp cCx1v5/FfaQ/IQHujqtG9Q 6121d11f9abfa6b1b36330eafaa2196249a9c92f989be25c9fac1558292c920f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.023 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'hchen', 'ovpn': 'hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hchen", "ovpn": "hchen@host12 hUr3k0rugStZMjvIxIvCOg 9d57e14d49901f18b24ee4076ae7e6a2f9eb6fd9fbce786660c448486c966fca"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.023 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jcollin', 'ovpn': 'jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jcollin", "ovpn": "jcollin@earth +H4Hk4WcNuqdQj7ch/Nulw 8426545e6457c9e1e8adca2af5ddf836fbcfb433cdc5359fd135afdf4e0f7d2a\njcollin@stratocaster jbjV3FsrsTJwyKUA3Y8VVQ 0439745f795fef1399636bd550040d45445d1607b471284c5c9b9dbccc86a987\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.034 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'xxg', 'ovpn': 'xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xxg", "ovpn": "xxg@zte Y2d/Ov201XMivDNwo4nUoQ 5e5da8d579793601699af628300430c1e5dd469c8bcff7c3ee11d23ec004bdcc"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.045 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pcuzner', 'ovpn': 'pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pcuzner", "ovpn": "pcuzner@rh460p oK28wU5DSabvEL4VjDRhEg a449ed81d7e2970f418263fb3ce10dd711d03925a0990ddf298f826aae1caa53"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.056 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mkogan', 'ovpn': 'mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mkogan", "ovpn": "mkogan@mkP50 f/LENRJbsyepmvZA23F7Fg b908f1c0237a7ee56b73dc42f2df79b49ca83d6f4573f5229e7cfe6b4ad7b6a2"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.056 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vumrao', 'ovpn': 'vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vumrao", "ovpn": "vumrao@redhat TMNqzMvbJS8Va/8nT9QUQw ab386c2bd7c6796d5413e4d841a16dda2504cca6d95df831a652a30d2e5655ed"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.067 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'henrix', 'ovpn': 'henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "henrix", "ovpn": "henrix@hermes iPPDBfnLzP5Pe5FuTcJBmw b26aefb8a61451066f984e074f708ea9ca6b2c5d7cca35996c08b0b2bb2c2736"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.068 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pbs1108', 'ovpn': 'pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pbs1108", "ovpn": "pbs1108@bspark_cli0 nSCINdeTLTLEO5JP/GIwRQ 76372ad6f7ad731556ff13605c3729eacaf59dcf7f9ac82dd9a8501bd95d3b26"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.079 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'clacroix', 'ovpn': 'clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "clacroix", "ovpn": "clacroix@redhat.com ZGY9sgvjT0BuJRi9zrULzg c3311aba4025aa42cd78c999dcee4e2c16415a3ac44ac8c95b77838459ef3315"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.089 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'epuertat', 'ovpn': 'epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "epuertat", "ovpn": "epuertat@private BnUoirwxGRWXtLxulJU5xA da2cfc4995bed82ef85db3633edad0a7eb2c32ba559a48259b10be94a8fdf006"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.090 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'tdehler', 'ovpn': 'tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "tdehler", "ovpn": "tdehler@think 7e0WC4Vh86XWZviZ9WBMgw 4dc8477db6e4f40312e6b2b9db293dc009e49e518015ace20431c0fb69025461"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.101 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'laura', 'ovpn': 'laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "laura", "ovpn": "laura@flab 2DGHIAxD39eNKOPkn3M67w ab1ae304abed3824a68b5c0ecf4f92fca76a4f8b9fcbcc0ca43388a85b7f9305"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.112 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'yaarit', 'ovpn': 'yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yaarit", "ovpn": "yaarit@centos TBuWjkAsj1GB/V9eWc/R1Q 7bd86a857dec48dc25850ecf0c00486d9a89c2ff5f88b2f28c3e36bdeb139fce"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.123 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rpavani1998', 'ovpn': 'admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rpavani1998", "ovpn": "admin1@rajula 31GbDo9d1YnW5BQ8u3utvw a2da13cb840f848846023c85442ba7bcce97dc186056a0ecc036a220d7eb7fc3"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.123 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rishabh', 'ovpn': 'rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rishabh", "ovpn": "rishabh@p50 zdJ4XsBdVugwMrqJOSBi3Q c78bb28ba5cf2bf9c8edb80fe57814d60cd2ffdbd874cf9a271e5adf171bb0c4"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.134 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'skrah', 'ovpn': 'skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "skrah", "ovpn": "skrah@thinkpad IboPmnUdsLqqRXlHQ1RT5w bf85db9e916dceaf84a1e6ea33c59eb4adb424cb4e727ce0a903a3498b799ed2"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.145 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'smanjara', 'ovpn': 'smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "smanjara", "ovpn": "smanjara@fedora OfpUShUejOW2W/R4FlKQ7Q fab77ef619f730a5aaf255e4167aa7a35a7e5450d12820923d969036a6326b0f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.145 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'bengland', 'ovpn': 'bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "bengland", "ovpn": "bengland@bene-laptop ud1gWgoNggJTS7LQtPTZTA d3ebd084ce385cb450ce2f83c02dc66a1637dedbc7a8b191dab68acfc935af41"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.156 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pnawracay', 'ovpn': 'pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pnawracay", "ovpn": "pnawracay@flab LIurEXLa7xbr+jzf2wRJVg 04062ba385602b385fd17f14de3a0cad83c685b8078fd2f18cc9ad77a4f4762d"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.166 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'alfonsomthd', 'ovpn': 'alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "alfonsomthd", "ovpn": "alfonsomthd@localhost iG3hmp+gS/4aOL4Vp/Nj4w ef1480c222cb2d540cf932ef2f27967891b768c2ad90b59a12a5279ceaa18b47"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.167 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'oliveiradan', 'ovpn': 'oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "oliveiradan", "ovpn": "oliveiradan@opensuse-dev twDqMYwYsdYmbgyCpUnElw ec3ee80ddc747c3ca5e1455a122279f8e1e642c5c09aa9c2ca7fec142f55089e"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.178 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'swagner', 'ovpn': 'swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "swagner", "ovpn": "swagner@ubuntu 64V1h0Se0FmBQNH7KLibbQ ad7c91e9e2f7f3999492d5e41fbbc993327d37929bd09606227367d75e5556ba"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.188 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'yuvalif', 'ovpn': 'ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "yuvalif", "ovpn": "ylifshit@localhost dyc2NU2pMz8NF8/uR2kMxA 3a6f1f9e55b5116f74d01ffbabdc339054088d257a16cf9fafcfe05b27fa678e\nyuvalif@medea gy0T/Hj/6DnVSKyOKD0/ag 468fe695fad8610a8f2eba6d1015949e4e6dc3d62c50ecf03579928d45a8d2ae\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.190 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kkeithle', 'ovpn': 'kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kkeithle", "ovpn": "kkeithle@kkeithle.usersys.redhat.com FPnVevv1sp5hlWoJeDCe/g e5a1fa7ccf678b91ed570983d5420c98f109d507442c8e4dcd50803e0d71c852"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.200 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'emmericp', 'ovpn': 'paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "emmericp", "ovpn": "paul.emmerich@croit.io RN4hOorvA7irUg/3ViM9KQ 3bd06194186d2624cadf255fa1c38ddf7dded0a6d83dc6001cd55fcc0a899130"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.211 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mchangir', 'ovpn': 'mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mchangir", "ovpn": "mchangir@indraprastha 1p71eDTdLMzkySFg0d5qEg bc4fda0ac9ef5cfbfab0cd9e5ed956c75b6280a04043c90c288e62b416b42da0"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.222 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sidharthanup', 'ovpn': 'sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sidharthanup", "ovpn": "sidharthanup@strawberryfields IZq91vqA+RG1Rtn3JFZb6Q a2873481cac9b8b4a0bd8bebe0248b3dccb370dd18b56a4dae713ca1fb0c4286"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.223 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'varsha', 'ovpn': 'varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "varsha", "ovpn": "varsha@local AGrJViO6wZF253aFeh/r5w f1248029ccdc45fd6e8fddb09ab3f8aa01123ab69cc6f3f11273528ce33e5a31"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.233 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sjust', 'ovpn': 'sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sjust", "ovpn": "sjust@pondermatic sCd6606QpID1PnHn0AxFag 46da7d4c77cb1238f83a34f99e774707110f997d88f3a0fd240aac9b7b7bbc85\nsjust@rex004 YS0in6YtQJHx5aUo7ZHi8Q bdd5977d05171a365539b19fae283ec2e3c7389516664692b9bbbaf98c7b61f4\nsjust@office w19UilyC/xu7uCzv0DnWRg ab20efc114b769bf4c2cf313eb30db09c2e2f8234992f120cfc3d1b8b347ed3c\nsam@deepthought 44sCi+GEfY0zjKo5M/4FiQ ed1eedd14ca68116a2000477fa078f8f736d0a15640723c32204bb30f14cb888\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.244 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ideepika', 'ovpn': 'deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ideepika", "ovpn": "deepika@x1carbon hyRj1/9UjWhVzhav5+LNKA e3f7d2882c2b1a410dbb5458c08f280814d652e34c87e96f8a2e8bef1928926f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.245 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'gsalomon', 'ovpn': 'gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gsalomon", "ovpn": "gsalomon@home BrlCVxStNlEZGf5yYYAz1w 4b0c8d5b57dae1328c16d3017b59b632ccdfebe4135209fa97748c70ff00cc46"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.255 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'soumyakoduri', 'ovpn': 'skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "soumyakoduri", "ovpn": "skoduri@localhost QwKbKHfNAOenVV0mkO8NXw 40d74abb20ebba9eaca5e58f6c821670d23f5db5405fc6e3072b985a397c0104"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.266 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kyr', 'ovpn': 'kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kyr", "ovpn": "kyr@suse Xu4+Yi98Il4ETOAav6okqA 03bc46e5fac6346cd82ff681f756860f98e0c61168633ce23325efde11a1964a"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.267 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sseshasa', 'ovpn': 'sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sseshasa", "ovpn": "sseshasa@thinkpad jyB1pr0I3qsDknkTnJTMjg 72ac1456e344c22fd940d0ba0e035aa3819ef7cd3891e53308aa92ba2dec8849"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.278 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rfriedma', 'ovpn': 'rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rfriedma", "ovpn": "rfriedma@rflap.redhat.com 5+OUPoyz8K0M0kcymdQOjA 40ce705001f31d7156c965228938cd4b02ae1a2c43dac1bbcd1b538e70312189"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.288 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'lmb', 'ovpn': 'lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "lmb", "ovpn": "lmb@hermes LMS8kAikL0iqw2S6IbXa3Q f57a493b31e7ed02a2563dd4295278d4842dc698b4c635d011a8d2b4b1fd5c2b"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.289 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ksirivad', 'ovpn': 'ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ksirivad", "ovpn": "ksirivad@ksirivad prZzE08FqnY6hFtGk0Q6XQ 2ef0878b0050cf28775813fe0991f9a746c07e61920280ce29ee69088eda5efc\nksirivad@macbook 0ivvJOZVAywT15aJCDF2xw 0066940a99b2d37dcd856c8622f48e0d8cb59c092f8f96cc2352ec0bc979ce45\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.300 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ofriedma', 'ovpn': 'ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ofriedma", "ovpn": "ofriedma@myhost IIifKuFYfX5EqO3mhV8lAg 82cb23a98361f3624b2f1f4e0f694f9439ae7735b171ba5e3bb5f0744f5ddfc4"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.310 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'h4iresis', 'ovpn': 'h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "h4iresis", "ovpn": "h4iresis@fedora NyMdx+dAfmujNSDB63kI+g bcf6c36594585e0fbf9f037bf1a13fe63db4b82a52ff1349349fac748b5c4e52"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.311 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyukri', 'ovpn': 'shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyukri", "ovpn": "shyukri@t480 KEbvWFIOWqkHsLgffi6GMQ fcd405d0cf6b2c7ba53397c4ed1156e852317fb02bcd6f0e98e5eb3c1c3aa4b5"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.321 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'scarvalhojr', 'ovpn': 'scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "scarvalhojr", "ovpn": "scarvalhojr@milestones utvFbB+X1G75M589jLfFXQ 9c62bf416c9204b25028ba0e5bd45f754c4aa8f84a85b9d5ded02aa83f03e82e"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.332 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rakesh', 'ovpn': 'rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rakesh", "ovpn": "rakesh@fedora CXsWP7Wzu2C+bTN6mIeMiw 2f4f492cd085f1f46d1d5018aeb7705ba855c4481e230a8188b8f5ea9994b91b"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.333 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'ryg-', 'ovpn': 'ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "ryg-", "ovpn": "ryg@wombat Y8asVyh6Qu+EEmWlRd1zXA d4923cf7be8e59728ea290aaec7670903b39b6444e5d303016338ddf5ab7837f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.343 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'prsrivas', 'ovpn': 'prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prsrivas", "ovpn": "prsrivas@t480 dvGwTZM2FivQPrMGmkY53g 80afabbb654af66ec01463d0e0425be5643eaca10bbf756f2b8313645d2b8da3"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.354 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'zdover', 'ovpn': 'zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "zdover", "ovpn": "zdover@lenovot5301 88ArNhsAMXq/nkSC9tKDig 131b8214bbea52345220522f8dd13c768bc5c1e41e4d164e7138719cfc3295d3"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.365 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'rraja', 'ovpn': 'rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "rraja", "ovpn": "rraja@p50 4NMUghtas9gI46n7X0iJXA cc646e47c8b503eb38d30efa47e459076af52b4fc001c4959833f32ddc563a02"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.366 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'toabctl', 'ovpn': 'toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "toabctl", "ovpn": "toabctl@minze a3I/7JakMEEANsNXCYM5XA b9715003902937328801993a359da43c61ab0be8d17f92371a48dca0821fb13f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.376 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mgfritch', 'ovpn': 'mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mgfritch", "ovpn": "mgfritch@laptop 94prpsaAkBwTzMmF9JR4WA 3adb8aad97b1f78c74d3ae91f21f4b165749843dacabad74f0039212c5a5db39"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.387 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'xiubli', 'ovpn': 'root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xiubli", "ovpn": "root@lxbceph0 /BNUvfmLPqzfbIz/Jzlqkw 3e109dcb826d45a6ee7e8195fc025b257b3d369e135400bf0a4bf503d48503e3\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.387 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jafaj', 'ovpn': 'jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jafaj", "ovpn": "jan@jfsuselaptop ddAKCkWSc1rEhQadU15jkA 90ddc588ca94ab776b539a1c5246a216dfbebabba3709ae803d386f978b65c6f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.398 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jschmid', 'ovpn': 'jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jschmid", "ovpn": "jschmid@zulu BqyepNoWzZbbsslJbvCuYg 4ce9b431fdd8a06856cfe00b88665da11dde1cacd72d40bf3dbe54ff633066ef"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.409 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kachau', 'ovpn': 'kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kachau", "ovpn": "kachau@thinkpad dX7+Zp6Z+rMSZTbxL52c7g 59d0d7be858e4a3769eaca4563399989298004a271f60b08cc36bf4130e089fe"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.410 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'khiremat', 'ovpn': 'khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "khiremat", "ovpn": "khiremat@kotresh-T490s 7o9JjenDm1h34Zmd2SWO2g 791981cd84bd4c80bf3ff6076ebc64b3ebf8432903e36d1a49bbee27561f8315"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.420 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kapandya', 'ovpn': 'kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kapandya", "ovpn": "kapandya@thinkpad g4JoonWShBhyS6fEdaHh7A 2990a987bf7afee6e3c8b5236b3dac813a57dbdce47ec63965c40b1b7bdafd67"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.431 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'prasad', 'ovpn': 'prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prasad", "ovpn": "prasad@macprasad tJYuW/3nDX8ikaLyJQdpAQ 1544f87809a170bb61069b588630467b05283a1347945b9545205b0a28110f89"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.432 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vikhyat', 'ovpn': 'vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vikhyat", "ovpn": "vikhyat@redhat ae77Y7dK1BxLEfQvAOJo8A a62d65a2aa09a75dca299ce0424caf6bcbc52696b22aa1567432aac34d78f18f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.443 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'shraddhaag', 'ovpn': 'shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shraddhaag", "ovpn": "shraddhaag@asus w1lmx5nwk8QQ/4u3UfGsJQ 98c5840f994a5e35b234375b07b1782f9a92bdcd5cf73e5847027045ab88e6d2"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.454 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'gkyratsas', 'ovpn': 'gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "gkyratsas", "ovpn": "gkyratsas@suse.com PO5Ut+2Z4kCMjaumOyA+1A 70fafe30ccb801c8c0a059a7eb3a842a2d08015b828c63bc1a702caf056014c7"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.465 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'prateek', 'ovpn': 'prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "prateek", "ovpn": "prateek@linux kj4pU4yLf+76XxY0mvoIwg c24292e824e3e43ebf7465f1079449ce80bf26311426ae3914cdf1fe8d591c67"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.465 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jelopez', 'ovpn': 'jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jelopez", "ovpn": "jelopez@imac XbzY2mjr57lSPExJp0ZBgA 4f5dfad692c25293cd58393bd8ddae91b4dc0b2561c860072e0665933531208e"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.476 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dcain', 'ovpn': 'dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dcain", "ovpn": "dcain@fullsteam 7vptxC1ZWCXLUV3FJwWQAg 2d73728e16c5f724e5f28eb326c24b8e13e7b361bcbaf506809f539e80223389"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.476 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'sunny', 'ovpn': 'sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "sunny", "ovpn": "sunny@devBox 81r/AKNOWhvaD0ESYy6tTg 58b86902f858bc3f1edb12cfc41ce10ed163985db2f7c253de08b7f1c310fe11"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.487 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'xuxuehan', 'ovpn': 'xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "xuxuehan", "ovpn": "xuxuehan@xuxuehan-PC aJDFkJ1+XzbovMyvRiFCAQ 5a71908d9bc8372ce0fff3bdbce4ea0e27eb11af0ed30e573f93d7a0b4335279\nxuxuehan@xuxuehan-notebook LzP6obPSYuao0iCJLQ/92A 3af116b690da581c77912a0c5fe184c84ca78526f67331db15714bc4ffddbcdc\n"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.498 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'aclewett', 'ovpn': 'aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "aclewett", "ovpn": "aclewett@mac bP0KgOkhW1GxTiugD3jMQA 90009b65da8175e083ee0e79d4953268f44493b98c2b532854da8d9e1d14e798"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.509 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'cblum', 'ovpn': 'cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "cblum", "ovpn": "cblum@ ad4mZ9Cg0u6H8AdARvQUZw 4e3cec570cbe42786e5190de4194c2937f1073aef62a25b61bd85e05ee23db56"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.509 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mlatimer', 'ovpn': 'mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mlatimer", "ovpn": "mlatimer@lx5510 3lc4nP664Dmg63yr8yKG7w 3778fb16fb62d1a7d61f41cf2c2e72fd9acaae2805de2dc34497ef41ee1ad64f"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.520 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'shyamsundarr', 'ovpn': 'shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shyamsundarr", "ovpn": "shyamsundarr@bartimaeus HUEzMC00f5f/tdydddWGNw e9032244bda086ecca6abc4f525296f9a125e4bbdb768d8c76c6d53d02f475d5"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.531 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'shuzhenyi', 'ovpn': 'bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "shuzhenyi", "ovpn": "bigwhite@wrjj37deMacBook-Pro-2.local VoTYmzsOPazeWcH+DOHXKw d7502e96b37ebc2ee5e0c14c6026f10e2e72b49be70dd1c5c7e9751fafcd8501"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.531 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'mdipalma', 'ovpn': 'mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "mdipalma", "ovpn": "mdipalma@mdmac IVn9Oy/SxZEmDgSWSeo3jA ea4dc08cb39ba55ade357c4949bf3f7d4d0322171ff5b5dac3c06d47cdc38f14"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.542 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'kchang', 'ovpn': 'kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "kchang", "ovpn": "kchang@kiefer-uchi Vy8tg3hSizdX5fIAJFgG/A 912a49a77b5722fdf4b9e304ee329c7b808728d73bb1e1f70cf55bd1221a3da3"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.553 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'vereddy', 'ovpn': 'vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "vereddy", "ovpn": "vereddy@vereddy.remote.csb 8jHiMDBfgQxu/8+MTlPR1A 07898add1a48c50647de4a81ad23db0c8545202cb56d3172db9b22df32c6b4a2"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.554 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'benhanokh', 'ovpn': 'gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "benhanokh", "ovpn": "gbenhano@localhost.localdomain uMf63iCCcqDt1rUnOSujaw 8c554c50bf8e27d3373515168e59f4ed3e4c48bace5689838268569948545ed5"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.564 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jsalomon', 'ovpn': 'jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jsalomon", "ovpn": "jsalomon@Josh-laptop 9V9NVwrt57LxXiqRh5Q7PQ 4cf0b0dfa39cf1e72f51bbfed19b4d88bdb286d09e54a99eb35721e33b1ae7b0"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.575 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'jolmomar', 'ovpn': 'jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "jolmomar", "ovpn": "jolmomar@thinkpad CnoIzb/786lZh1NINj0xAA e60eeb560654da4b84977a4f9dbccd599bf9460980985ee5726595eb4a46e640"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.577 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'hmunjulu', 'ovpn': 'hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "hmunjulu", "ovpn": "hmunjulu@thinkpad +5kJJIhXveJw6Q8ytILUSA 546f32fb058ff40ef3113e6c258b0f1cba6611155ba0a40a55646946cdb53019"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.586 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'julpark', 'ovpn': 'julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "julpark", "ovpn": "julpark@thinkpad svj+nOCQG95zuBUwdJoluQ 7252f05ef02b16fd74d931d6cf937304afbb098dbdbde13d377e884fe35e2fb8"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.597 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'pdhange', 'ovpn': 'prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "pdhange", "ovpn": "prashant@cephdev JZCEN2T5TXiHnU5yLG4hBg 3f94ec1bd179dafd2d58335ac4bea35808d394d30fa9e174fe16d6b872450fe0"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.608 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'dpivonka', 'ovpn': 'dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "dpivonka", "ovpn": "dpivonka@thinkpad huZBxlH4VpGff0Q68m/2Rg 43cc9c90a55fa3395862e679b60c5e6c016a59f186e7e8798579cd6046cc4ec8"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.609 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'adking', 'ovpn': 'adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "adking", "ovpn": "adking@laptop1 VkjYUD1hJpR8sFhrH6MIcg 28c610bc9f7321de315923288d24650bdb4bc90593b423f1f59db16d67b6d625"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.619 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'amathuri', 'ovpn': 'amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "amathuri", "ovpn": "amathuri@thinkpad WDHZL4VspBmlz5dzRlAGUQ 7f2c5d8e2554b869a09fdc20d6e4aabffa8200ef8e2b8d1bc951bfc673757267"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.630 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'name': 'nizamial09', 'ovpn': 'nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99'}) => {"ansible_loop_var": "item", "changed": false, "item": {"name": "nizamial09", "ovpn": "nia@redhat +vB2tfopNqiNnGugVmX3ew f9f6878bb5e6bb2353fa8da7d76e2d621e9fbf89a4c7583d7265cb984e231a99"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.681 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:50:55 +0000 (0:00:01.796) 0:01:09.072 ******* 2021-02-12T17:50:55.707 INFO:teuthology.task.ansible.out: TASK [users : Filter the revoked_users list] *********************************** 2021-02-12T17:50:55.708 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:50:55.741 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:50:55 +0000 (0:00:00.059) 0:01:09.132 ******* 2021-02-12T17:50:55.965 INFO:teuthology.task.ansible.out: TASK [users : Remove revoked users] ******************************************** 2021-02-12T17:50:55.966 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=zcerza) => {"ansible_loop_var": "item", "changed": false, "item": "zcerza", "name": "zcerza", "state": "absent"} 2021-02-12T17:50:56.152 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=aschoen) => {"ansible_loop_var": "item", "changed": false, "item": "aschoen", "name": "aschoen", "state": "absent"} 2021-02-12T17:50:56.338 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=andrew) => {"ansible_loop_var": "item", "changed": false, "item": "andrew", "name": "andrew", "state": "absent"} 2021-02-12T17:50:56.524 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=sweil) => {"ansible_loop_var": "item", "changed": false, "item": "sweil", "name": "sweil", "state": "absent"} 2021-02-12T17:50:56.711 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=brad) => {"ansible_loop_var": "item", "changed": false, "item": "brad", "name": "brad", "state": "absent"} 2021-02-12T17:50:56.898 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=kefu) => {"ansible_loop_var": "item", "changed": false, "item": "kefu", "name": "kefu", "state": "absent"} 2021-02-12T17:50:57.084 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=shylesh) => {"ansible_loop_var": "item", "changed": false, "item": "shylesh", "name": "shylesh", "state": "absent"} 2021-02-12T17:50:57.270 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=gmeno) => {"ansible_loop_var": "item", "changed": false, "item": "gmeno", "name": "gmeno", "state": "absent"} 2021-02-12T17:50:57.457 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=alfredodeza) => {"ansible_loop_var": "item", "changed": false, "item": "alfredodeza", "name": "alfredodeza", "state": "absent"} 2021-02-12T17:50:57.643 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=trhoden) => {"ansible_loop_var": "item", "changed": false, "item": "trhoden", "name": "trhoden", "state": "absent"} 2021-02-12T17:50:57.819 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=nishtha) => {"ansible_loop_var": "item", "changed": false, "item": "nishtha", "name": "nishtha", "state": "absent"} 2021-02-12T17:50:58.006 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=yguang) => {"ansible_loop_var": "item", "changed": false, "item": "yguang", "name": "yguang", "state": "absent"} 2021-02-12T17:50:58.202 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=sdieffen) => {"ansible_loop_var": "item", "changed": false, "item": "sdieffen", "name": "sdieffen", "state": "absent"} 2021-02-12T17:50:58.388 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=brian) => {"ansible_loop_var": "item", "changed": false, "item": "brian", "name": "brian", "state": "absent"} 2021-02-12T17:50:58.574 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=pmcgarry) => {"ansible_loop_var": "item", "changed": false, "item": "pmcgarry", "name": "pmcgarry", "state": "absent"} 2021-02-12T17:50:58.749 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=karnan) => {"ansible_loop_var": "item", "changed": false, "item": "karnan", "name": "karnan", "state": "absent"} 2021-02-12T17:50:58.936 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=ryneli) => {"ansible_loop_var": "item", "changed": false, "item": "ryneli", "name": "ryneli", "state": "absent"} 2021-02-12T17:50:59.122 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=dlambrig) => {"ansible_loop_var": "item", "changed": false, "item": "dlambrig", "name": "dlambrig", "state": "absent"} 2021-02-12T17:50:59.307 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=icolle) => {"ansible_loop_var": "item", "changed": false, "item": "icolle", "name": "icolle", "state": "absent"} 2021-02-12T17:50:59.494 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=soumya) => {"ansible_loop_var": "item", "changed": false, "item": "soumya", "name": "soumya", "state": "absent"} 2021-02-12T17:50:59.679 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=jspray) => {"ansible_loop_var": "item", "changed": false, "item": "jspray", "name": "jspray", "state": "absent"} 2021-02-12T17:50:59.865 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=loic) => {"ansible_loop_var": "item", "changed": false, "item": "loic", "name": "loic", "state": "absent"} 2021-02-12T17:51:00.051 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=erwan) => {"ansible_loop_var": "item", "changed": false, "item": "erwan", "name": "erwan", "state": "absent"} 2021-02-12T17:51:00.227 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=jj) => {"ansible_loop_var": "item", "changed": false, "item": "jj", "name": "jj", "state": "absent"} 2021-02-12T17:51:00.413 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=amarangone) => {"ansible_loop_var": "item", "changed": false, "item": "amarangone", "name": "amarangone", "state": "absent"} 2021-02-12T17:51:00.610 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=oprypin) => {"ansible_loop_var": "item", "changed": false, "item": "oprypin", "name": "oprypin", "state": "absent"} 2021-02-12T17:51:00.786 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=adamyanova) => {"ansible_loop_var": "item", "changed": false, "item": "adamyanova", "name": "adamyanova", "state": "absent"} 2021-02-12T17:51:00.972 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=sbillah) => {"ansible_loop_var": "item", "changed": false, "item": "sbillah", "name": "sbillah", "state": "absent"} 2021-02-12T17:51:01.158 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=onyb) => {"ansible_loop_var": "item", "changed": false, "item": "onyb", "name": "onyb", "state": "absent"} 2021-02-12T17:51:01.344 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=jwilliamson) => {"ansible_loop_var": "item", "changed": false, "item": "jwilliamson", "name": "jwilliamson", "state": "absent"} 2021-02-12T17:51:01.530 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=kmroz) => {"ansible_loop_var": "item", "changed": false, "item": "kmroz", "name": "kmroz", "state": "absent"} 2021-02-12T17:51:01.717 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=shehbazj) => {"ansible_loop_var": "item", "changed": false, "item": "shehbazj", "name": "shehbazj", "state": "absent"} 2021-02-12T17:51:01.903 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=abhishekvrshny) => {"ansible_loop_var": "item", "changed": false, "item": "abhishekvrshny", "name": "abhishekvrshny", "state": "absent"} 2021-02-12T17:51:02.090 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=asheplyakov) => {"ansible_loop_var": "item", "changed": false, "item": "asheplyakov", "name": "asheplyakov", "state": "absent"} 2021-02-12T17:51:02.278 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=liupan) => {"ansible_loop_var": "item", "changed": false, "item": "liupan", "name": "liupan", "state": "absent"} 2021-02-12T17:51:02.465 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=adeza) => {"ansible_loop_var": "item", "changed": false, "item": "adeza", "name": "adeza", "state": "absent"} 2021-02-12T17:51:02.653 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=pranith) => {"ansible_loop_var": "item", "changed": false, "item": "pranith", "name": "pranith", "state": "absent"} 2021-02-12T17:51:03.957 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item=dorinda) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "dorinda", "name": "dorinda", "remove": false, "state": "absent"} 2021-02-12T17:51:05.277 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item=zyan) => {"ansible_loop_var": "item", "changed": true, "force": false, "item": "zyan", "name": "zyan", "remove": false, "state": "absent"} 2021-02-12T17:51:05.473 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=nlevine) => {"ansible_loop_var": "item", "changed": false, "item": "nlevine", "name": "nlevine", "state": "absent"} 2021-02-12T17:51:05.660 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=tbrekke) => {"ansible_loop_var": "item", "changed": false, "item": "tbrekke", "name": "tbrekke", "state": "absent"} 2021-02-12T17:51:05.836 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=taco) => {"ansible_loop_var": "item", "changed": false, "item": "taco", "name": "taco", "state": "absent"} 2021-02-12T17:51:06.023 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=louis) => {"ansible_loop_var": "item", "changed": false, "item": "louis", "name": "louis", "state": "absent"} 2021-02-12T17:51:06.209 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=amarango) => {"ansible_loop_var": "item", "changed": false, "item": "amarango", "name": "amarango", "state": "absent"} 2021-02-12T17:51:06.395 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=oobe) => {"ansible_loop_var": "item", "changed": false, "item": "oobe", "name": "oobe", "state": "absent"} 2021-02-12T17:51:06.581 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=rturk) => {"ansible_loop_var": "item", "changed": false, "item": "rturk", "name": "rturk", "state": "absent"} 2021-02-12T17:51:06.766 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=fche) => {"ansible_loop_var": "item", "changed": false, "item": "fche", "name": "fche", "state": "absent"} 2021-02-12T17:51:06.952 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=jbainbri) => {"ansible_loop_var": "item", "changed": false, "item": "jbainbri", "name": "jbainbri", "state": "absent"} 2021-02-12T17:51:07.149 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=kdhananj) => {"ansible_loop_var": "item", "changed": false, "item": "kdhananj", "name": "kdhananj", "state": "absent"} 2021-02-12T17:51:07.202 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:07 +0000 (0:00:11.461) 0:01:20.593 ******* 2021-02-12T17:51:07.703 INFO:teuthology.task.ansible.out: TASK [users : Write /keys-repo-sha1 sentinel file] ***************************** 2021-02-12T17:51:07.705 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "checksum": "5ebc9fcf8859a9c716ab10089a7bec7e44d29368", "dest": "/keys-repo-sha1", "gid": 0, "group": "root", "md5sum": "ee06ed012f51c2e3764171d4b9180349", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_runtime_t:s0", "size": 40, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613152267.223534-192173275002117/source", "state": "file", "uid": 0} 2021-02-12T17:51:07.737 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:07 +0000 (0:00:00.535) 0:01:21.129 ******* 2021-02-12T17:51:07.763 INFO:teuthology.task.ansible.out: TASK [common : Log the OS name, version and release] *************************** 2021-02-12T17:51:07.765 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => { "msg": "Host smithi107.front.sepia.ceph.com is running CentOS 8.2 (Core)" } 2021-02-12T17:51:07.797 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:07 +0000 (0:00:00.059) 0:01:21.188 ******* 2021-02-12T17:51:07.834 INFO:teuthology.task.ansible.out: TASK [common : Including major version specific variables.] ******************** 2021-02-12T17:51:07.835 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml) => {"ansible_facts": {"nrpe_selinux_packages": ["python3-libsemanage", "python3-policycoreutils"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/centos_8.yml"} 2021-02-12T17:51:07.866 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:07 +0000 (0:00:00.069) 0:01:21.257 ******* 2021-02-12T17:51:07.892 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 6)] *********************** 2021-02-12T17:51:07.893 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:07.924 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:07 +0000 (0:00:00.057) 0:01:21.315 ******* 2021-02-12T17:51:08.424 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone (RHEL/CentOS 7)] *********************** 2021-02-12T17:51:08.426 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "cmd": "timedatectl | grep -E \"Time ?zone\" | sed -e \"s/.*: \\(.*\\) (.*/\\1/\"", "delta": "0:00:00.310296", "end": "2021-02-12 17:51:08.401677", "rc": 0, "start": "2021-02-12 17:51:08.091381", "stderr": "", "stderr_lines": [], "stdout": "Etc/UTC", "stdout_lines": ["Etc/UTC"]} 2021-02-12T17:51:08.458 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.534) 0:01:21.850 ******* 2021-02-12T17:51:08.484 INFO:teuthology.task.ansible.out: TASK [common : Reboot RHEL7 to workaround systemd bug] ************************* 2021-02-12T17:51:08.485 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:08.518 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.059) 0:01:21.909 ******* 2021-02-12T17:51:08.544 INFO:teuthology.task.ansible.out: TASK [common : Wait for reboot in case of systemd workaround] ****************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:08.583 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.064) 0:01:21.974 ******* 2021-02-12T17:51:08.609 INFO:teuthology.task.ansible.out: TASK [common : Set /etc/localtime (RHEL/CentOS 6)] ***************************** 2021-02-12T17:51:08.610 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:08.642 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.059) 0:01:22.033 ******* 2021-02-12T17:51:08.668 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone (RHEL/CentOS >= 7)] **************************** 2021-02-12T17:51:08.669 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:08.701 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.058) 0:01:22.092 ******* 2021-02-12T17:51:08.727 INFO:teuthology.task.ansible.out: TASK [common : Install yum on Fedora 22 and later] ***************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:08.760 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.059) 0:01:22.151 ******* 2021-02-12T17:51:08.797 INFO:teuthology.task.ansible.out: TASK [common : Set entitlements_path] ****************************************** 2021-02-12T17:51:08.798 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:08.832 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.071) 0:01:22.223 ******* 2021-02-12T17:51:08.868 INFO:teuthology.task.ansible.out: TASK [common : Include Red Hat encrypted variables.] *************************** 2021-02-12T17:51:08.869 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=None) => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-12T17:51:08.902 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.070) 0:01:22.293 ******* 2021-02-12T17:51:08.929 INFO:teuthology.task.ansible.out: TASK [common : Set have_entitlements] ****************************************** 2021-02-12T17:51:08.931 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:08.962 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:08 +0000 (0:00:00.059) 0:01:22.353 ******* 2021-02-12T17:51:08.988 INFO:teuthology.task.ansible.out: TASK [common : Install CA Cert from Satellite Server] ************************** 2021-02-12T17:51:08.989 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.021 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.058) 0:01:22.412 ******* 2021-02-12T17:51:09.047 INFO:teuthology.task.ansible.out: TASK [common : Determine if node is registered with subscription-manager.] ***** 2021-02-12T17:51:09.048 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-12T17:51:09.079 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.058) 0:01:22.471 ******* 2021-02-12T17:51:09.105 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we're already registered] **************** 2021-02-12T17:51:09.107 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.139 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.059) 0:01:22.530 ******* 2021-02-12T17:51:09.164 INFO:teuthology.task.ansible.out: TASK [common : Register with subscription-manager.] **************************** 2021-02-12T17:51:09.166 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-12T17:51:09.196 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.057) 0:01:22.588 ******* 2021-02-12T17:51:09.222 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered if we just registered] ********************** 2021-02-12T17:51:09.223 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.255 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.058) 0:01:22.646 ******* 2021-02-12T17:51:09.282 INFO:teuthology.task.ansible.out: TASK [common : List CDN releases available to system] ************************** 2021-02-12T17:51:09.283 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.314 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.059) 0:01:22.706 ******* 2021-02-12T17:51:09.341 INFO:teuthology.task.ansible.out: TASK [common : Unregister Beta/Alpha/RC system with subscription-manager] ****** 2021-02-12T17:51:09.342 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.374 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.059) 0:01:22.765 ******* 2021-02-12T17:51:09.399 INFO:teuthology.task.ansible.out: TASK [common : Set rhsm_registered to false if Beta/Alpha/RC release] ********** 2021-02-12T17:51:09.401 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.433 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.058) 0:01:22.824 ******* 2021-02-12T17:51:09.459 INFO:teuthology.task.ansible.out: TASK [common : Configure internal beta repos in /etc/yum.repos.d/] ************* skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-12T17:51:09.492 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.058) 0:01:22.883 ******* 2021-02-12T17:51:09.517 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-12T17:51:09.519 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.550 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.058) 0:01:22.941 ******* 2021-02-12T17:51:09.576 INFO:teuthology.task.ansible.out: TASK [common : Get list of enabled RHSM repos] ********************************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.609 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.059) 0:01:23.001 ******* 2021-02-12T17:51:09.636 INFO:teuthology.task.ansible.out: TASK [common : Store list of enabled repos] ************************************ 2021-02-12T17:51:09.638 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.669 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.059) 0:01:23.060 ******* 2021-02-12T17:51:09.705 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos false if entitlements are missing or if we unregistered] *** 2021-02-12T17:51:09.706 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.741 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.072) 0:01:23.132 ******* 2021-02-12T17:51:09.767 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if rhsm_repos differs from repo_list] **** 2021-02-12T17:51:09.768 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.801 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.059) 0:01:23.192 ******* 2021-02-12T17:51:09.826 INFO:teuthology.task.ansible.out: TASK [common : Set replace_repos true if newly-subscribed] ********************* 2021-02-12T17:51:09.828 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.860 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.059) 0:01:23.251 ******* 2021-02-12T17:51:09.886 INFO:teuthology.task.ansible.out: TASK [common : Disable all rhsm repos] ***************************************** 2021-02-12T17:51:09.887 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} 2021-02-12T17:51:09.919 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.059) 0:01:23.310 ******* 2021-02-12T17:51:09.945 INFO:teuthology.task.ansible.out: TASK [common : Enable necessary rhsm repos] ************************************ 2021-02-12T17:51:09.946 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:09.979 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:09 +0000 (0:00:00.060) 0:01:23.371 ******* 2021-02-12T17:51:10.006 INFO:teuthology.task.ansible.out: TASK [common : Remove old apt-mirror repository definition.] ******************* 2021-02-12T17:51:10.008 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:10.039 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:10 +0000 (0:00:00.059) 0:01:23.430 ******* 2021-02-12T17:51:10.262 INFO:teuthology.task.ansible.out: TASK [common : Increase the yum timeout.] ************************************** 2021-02-12T17:51:10.263 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-12T17:51:10.296 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:10 +0000 (0:00:00.257) 0:01:23.687 ******* 2021-02-12T17:51:11.025 INFO:teuthology.task.ansible.out: TASK [common : Configure epel repos in /etc/yum.repos.d/] ********************** 2021-02-12T17:51:11.026 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'epel', 'value': {'name': 'Extra Packages for Enterprise Linux', 'mirrorlist': 'file:///etc/yum.repos.d/epel-mirrorlist', 'failovermethod': 'priority', 'enabled': '1', 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "86c0d58bde7ad4cfaafe592d2fefd92c920c42ca", "dest": "/etc/yum.repos.d/epel.repo", "gid": 0, "group": "root", "item": {"key": "epel", "value": {"enabled": "1", "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-mirrorlist", "name": "Extra Packages for Enterprise Linux"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 236, "state": "file", "uid": 0} 2021-02-12T17:51:11.449 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'epel-testing', 'value': {'name': 'Extra Packages for Enterprise Linux - Testing', 'mirrorlist': 'file:///etc/yum.repos.d/epel-testing-mirrorlist', 'failovermethod': 'priority', 'enabled': 0, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a1bad5509ed35b895c2061841236b1cfe10619ae", "dest": "/etc/yum.repos.d/epel-testing.repo", "gid": 0, "group": "root", "item": {"key": "epel-testing", "value": {"enabled": 0, "failovermethod": "priority", "gpgcheck": 0, "mirrorlist": "file:///etc/yum.repos.d/epel-testing-mirrorlist", "name": "Extra Packages for Enterprise Linux - Testing"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 262, "state": "file", "uid": 0} 2021-02-12T17:51:11.485 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:11 +0000 (0:00:01.189) 0:01:24.876 ******* 2021-02-12T17:51:11.958 INFO:teuthology.task.ansible.out: TASK [common : Configure local epel mirrorlists] ******************************* 2021-02-12T17:51:11.959 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=epel-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "3c9ba7bacd1e30bed014cefd1fb4eab0f0f8fa10", "dest": "/etc/yum.repos.d/epel-mirrorlist", "gid": 0, "group": "root", "item": "epel-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 484, "state": "file", "uid": 0} 2021-02-12T17:51:12.386 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=epel-testing-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "d542c3e2c30a0d3eb7d454775342c302c0ded15b", "dest": "/etc/yum.repos.d/epel-testing-mirrorlist", "gid": 0, "group": "root", "item": "epel-testing-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/epel-testing-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 540, "state": "file", "uid": 0} 2021-02-12T17:51:12.420 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.934) 0:01:25.811 ******* 2021-02-12T17:51:12.456 INFO:teuthology.task.ansible.out: TASK [common : Clean yum cache] ************************************************ 2021-02-12T17:51:12.458 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.490 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.070) 0:01:25.882 ******* 2021-02-12T17:51:12.516 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-12T17:51:12.518 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.549 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.059) 0:01:25.941 ******* 2021-02-12T17:51:12.575 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone in /etc/timezone] ****************************** 2021-02-12T17:51:12.577 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.608 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.058) 0:01:25.999 ******* 2021-02-12T17:51:12.633 INFO:teuthology.task.ansible.out: TASK [common : Inform the OS of the new timezone] ****************************** 2021-02-12T17:51:12.635 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.666 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.058) 0:01:26.058 ******* 2021-02-12T17:51:12.693 INFO:teuthology.task.ansible.out: TASK [common : Mask sleep units] *********************************************** 2021-02-12T17:51:12.694 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "sleep.target", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.704 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=suspend.target) => {"ansible_loop_var": "item", "changed": false, "item": "suspend.target", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.704 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=hibernate.target) => {"ansible_loop_var": "item", "changed": false, "item": "hibernate.target", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.706 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=hybrid-sleep.target) => {"ansible_loop_var": "item", "changed": false, "item": "hybrid-sleep.target", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.749 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.082) 0:01:26.140 ******* 2021-02-12T17:51:12.774 INFO:teuthology.task.ansible.out: TASK [common : Get the current timezone] *************************************** 2021-02-12T17:51:12.776 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.807 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.058) 0:01:26.198 ******* 2021-02-12T17:51:12.832 INFO:teuthology.task.ansible.out: TASK [common : Set the timezone] *********************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.865 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.058) 0:01:26.256 ******* 2021-02-12T17:51:12.890 INFO:teuthology.task.ansible.out: TASK [common : Add base OpenSUSE Leap repo] ************************************ 2021-02-12T17:51:12.891 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.922 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.057) 0:01:26.313 ******* 2021-02-12T17:51:12.948 INFO:teuthology.task.ansible.out: TASK [common : Add updates OpenSUSE Leap repo] ********************************* 2021-02-12T17:51:12.949 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:12.981 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:12 +0000 (0:00:00.059) 0:01:26.372 ******* 2021-02-12T17:51:13.007 INFO:teuthology.task.ansible.out: TASK [common : Refresh repos] ************************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:13.040 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:13 +0000 (0:00:00.059) 0:01:26.432 ******* 2021-02-12T17:51:13.519 INFO:teuthology.task.ansible.out: TASK [common : Set the hardware clock] ***************************************** 2021-02-12T17:51:14.564 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["hwclock", "--systohc"], "delta": "0:00:00.295621", "end": "2021-02-12 17:51:13.500342", "rc": 0, "start": "2021-02-12 17:51:13.204721", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} Friday 12 February 2021 17:51:13 +0000 (0:00:00.515) 0:01:26.947 ******* 2021-02-12T17:51:21.140 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (RedHat)] ***************************** 2021-02-12T17:51:21.141 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: krb5-workstation", "Installed: krb5-workstation-1.18.2-5.el8.x86_64", "Installed: krb5-libs-1.18.2-5.el8.x86_64", "Installed: libkadm5-1.18.2-5.el8.x86_64", "Removed: krb5-workstation-1.17-18.el8.x86_64", "Removed: libkadm5-1.17-18.el8.x86_64", "Removed: krb5-libs-1.17-18.el8.x86_64"]} 2021-02-12T17:51:21.174 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:21 +0000 (0:00:07.618) 0:01:34.566 ******* 2021-02-12T17:51:21.201 INFO:teuthology.task.ansible.out: TASK [common : Clean apt cache] ************************************************ 2021-02-12T17:51:21.202 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:21.234 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:21 +0000 (0:00:00.059) 0:01:34.625 ******* 2021-02-12T17:51:21.259 INFO:teuthology.task.ansible.out: TASK [common : Update apt cache] *********************************************** 2021-02-12T17:51:21.261 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:21.293 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:21 +0000 (0:00:00.059) 0:01:34.685 ******* 2021-02-12T17:51:21.319 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (Debian)] ***************************** 2021-02-12T17:51:21.321 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:21.354 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:21 +0000 (0:00:00.060) 0:01:34.745 ******* 2021-02-12T17:51:21.379 INFO:teuthology.task.ansible.out: TASK [common : Install Kerberos Packages (OpenSUSE Leap)] ********************** 2021-02-12T17:51:21.381 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:21.413 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:21 +0000 (0:00:00.059) 0:01:34.804 ******* 2021-02-12T17:51:21.882 INFO:teuthology.task.ansible.out: TASK [common : Add krb5 config file] ******************************************* 2021-02-12T17:51:21.884 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "b3c1a33d19a4e448c29374429629dff57657b576", "dest": "/etc/krb5.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/krb5.conf", "secontext": "system_u:object_r:krb5_conf_t:s0", "size": 131, "state": "file", "uid": 0} 2021-02-12T17:51:21.915 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:21 +0000 (0:00:00.502) 0:01:35.307 ******* 2021-02-12T17:51:25.220 INFO:teuthology.task.ansible.out: TASK [common : Make sure smartmontools is installed] *************************** 2021-02-12T17:51:25.221 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: smartmontools", "Installed: smartmontools-1:7.1-1.el8.x86_64", "Removed: smartmontools-1:6.6-3.el8.x86_64"]} 2021-02-12T17:51:25.253 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:25 +0000 (0:00:03.338) 0:01:38.645 ******* 2021-02-12T17:51:25.724 INFO:teuthology.task.ansible.out: TASK [common : Upload megacli and cli64 for raid monitoring and smart.pl to /usr/sbin/.] *** 2021-02-12T17:51:25.724 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=megacli) => {"ansible_loop_var": "item", "changed": false, "checksum": "cb3ef2aa6a3e3002b234f166e739a39b1f3a1bde", "dest": "/usr/sbin/megacli", "gid": 0, "group": "root", "item": "megacli", "mode": "0755", "owner": "root", "path": "/usr/sbin/megacli", "secontext": "system_u:object_r:bin_t:s0", "size": 2716736, "state": "file", "uid": 0} 2021-02-12T17:51:26.145 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=cli64) => {"ansible_loop_var": "item", "changed": false, "checksum": "a3b8773b14e8594ffbd9fc156c09436eb0184bc7", "dest": "/usr/sbin/cli64", "gid": 0, "group": "root", "item": "cli64", "mode": "0755", "owner": "root", "path": "/usr/sbin/cli64", "secontext": "system_u:object_r:bin_t:s0", "size": 1952038, "state": "file", "uid": 0} 2021-02-12T17:51:26.568 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=nvme) => {"ansible_loop_var": "item", "changed": false, "checksum": "98735dcacd6fec55dd92b67281b478603b7955b1", "dest": "/usr/sbin/nvme", "gid": 0, "group": "root", "item": "nvme", "mode": "0755", "owner": "root", "path": "/usr/sbin/nvme", "secontext": "system_u:object_r:bin_t:s0", "size": 580432, "state": "file", "uid": 0} 2021-02-12T17:51:26.603 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:26 +0000 (0:00:01.350) 0:01:39.995 ******* 2021-02-12T17:51:26.785 INFO:teuthology.task.ansible.out: TASK [common : Create /usr/libexec.] ******************************************* 2021-02-12T17:51:26.786 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/libexec", "secontext": "system_u:object_r:bin_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-12T17:51:26.818 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:26 +0000 (0:00:00.214) 0:01:40.210 ******* 2021-02-12T17:51:27.310 INFO:teuthology.task.ansible.out: TASK [common : Upload custom netsaint scripts for raid/disk/smart/monitoring to /usr/libexec/.] *** 2021-02-12T17:51:27.311 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item=smart.sh) => {"ansible_loop_var": "item", "changed": true, "checksum": "00d0404b545463be4a65f475e0870e07a48f35b6", "dest": "/usr/libexec/smart.sh", "gid": 0, "group": "root", "item": "smart.sh", "md5sum": "14041bf2666d8f8d277567958c7e9f69", "mode": "0755", "owner": "root", "secontext": "system_u:object_r:bin_t:s0", "size": 13355, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613152286.841033-213247812553879/source", "state": "file", "uid": 0} 2021-02-12T17:51:27.724 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=raid.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9d9a89805d8a9e4c7281934e887da31f7ac464fb", "dest": "/usr/libexec/raid.pl", "gid": 0, "group": "root", "item": "raid.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/raid.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 8602, "state": "file", "uid": 0} 2021-02-12T17:51:28.140 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=diskusage.pl) => {"ansible_loop_var": "item", "changed": false, "checksum": "9c74425885a83f23bc37817de46979ca5173ccb7", "dest": "/usr/libexec/diskusage.pl", "gid": 0, "group": "root", "item": "diskusage.pl", "mode": "0755", "owner": "root", "path": "/usr/libexec/diskusage.pl", "secontext": "system_u:object_r:bin_t:s0", "size": 3605, "state": "file", "uid": 0} 2021-02-12T17:51:28.175 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:28 +0000 (0:00:01.356) 0:01:41.567 ******* 2021-02-12T17:51:28.201 INFO:teuthology.task.ansible.out: TASK [common : Include apt_system vars] **************************************** 2021-02-12T17:51:28.202 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:28.234 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:28 +0000 (0:00:00.058) 0:01:41.625 ******* 2021-02-12T17:51:28.269 INFO:teuthology.task.ansible.out: TASK [common : Include yum_systems vars] *************************************** 2021-02-12T17:51:28.271 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"nagios_plugins_directory": "/usr/lib64/nagios/plugins", "nrpe_group": "nrpe", "nrpe_packages": ["nagios-common", "nrpe", "nagios-plugins", "nagios-plugins-load"], "nrpe_service_name": "nrpe", "nrpe_user": "nrpe"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/common/vars/yum_systems.yml"], "changed": false} 2021-02-12T17:51:28.303 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:28 +0000 (0:00:00.068) 0:01:41.694 ******* 2021-02-12T17:51:28.494 INFO:teuthology.task.ansible.out: TASK [common : Check for epel] ************************************************* 2021-02-12T17:51:28.495 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} 2021-02-12T17:51:28.528 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:28 +0000 (0:00:00.225) 0:01:41.920 ******* 2021-02-12T17:51:28.565 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe without epel] ************************************** 2021-02-12T17:51:28.566 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-common-4.0.8-2.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:28.576 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nrpe-2.15-7.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:28.587 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:28.597 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm) => {"ansible_loop_var": "item", "changed": false, "item": "http://apt-mirror.front.sepia.ceph.com/lab-extras/rhel7/x86_64/nagios-plugins-load-2.0.3-3.el7.x86_64.rpm", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:28.633 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:28 +0000 (0:00:00.105) 0:01:42.025 ******* 2021-02-12T17:51:29.640 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (RHEL/CentOS)] ************ 2021-02-12T17:51:29.641 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: nagios-common", "Installed: nrpe", "Installed: nagios-plugins", "Installed: nagios-plugins-load"]} 2021-02-12T17:51:29.674 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:29 +0000 (0:00:01.040) 0:01:43.065 ******* 2021-02-12T17:51:29.699 INFO:teuthology.task.ansible.out: TASK [common : Install nrpe package and dependencies (Ubuntu/Fedora)] ********** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:29.732 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:29 +0000 (0:00:00.058) 0:01:43.124 ******* 2021-02-12T17:51:30.173 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios sudoers.d for raid utilities.] ******************** 2021-02-12T17:51:30.175 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "b423b985ccbb69b65a711d372645a5f704b30e8b", "dest": "/etc/sudoers.d/90-nagios", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers.d/90-nagios", "secontext": "system_u:object_r:etc_t:s0", "size": 178, "state": "file", "uid": 0} 2021-02-12T17:51:30.206 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:30 +0000 (0:00:00.474) 0:01:43.598 ******* 2021-02-12T17:51:30.658 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios check_mem script] ********************************* 2021-02-12T17:51:30.660 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "39df629b10ded370443e2e4c84d690332c95104d", "dest": "/usr/lib64/nagios/plugins/check_mem.sh", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib64/nagios/plugins/check_mem.sh", "secontext": "system_u:object_r:nagios_unconfined_plugin_exec_t:s0", "size": 1964, "state": "file", "uid": 0} 2021-02-12T17:51:30.692 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:30 +0000 (0:00:00.485) 0:01:44.083 ******* 2021-02-12T17:51:30.728 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (Ubuntu)] ************************ 2021-02-12T17:51:30.730 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=DAEMON_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "DAEMON_OPTS", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=NRPE_OPTS) => {"ansible_loop_var": "item", "changed": false, "item": "NRPE_OPTS", "skip_reason": "Conditional result was False"} 2021-02-12T17:51:30.762 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:30 +0000 (0:00:00.070) 0:01:44.153 ******* 2021-02-12T17:51:30.964 INFO:teuthology.task.ansible.out: TASK [common : Configure nagios nrpe settings (RHEL/CentOS)] ******************* 2021-02-12T17:51:30.965 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-12T17:51:31.004 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:30 +0000 (0:00:00.242) 0:01:44.395 ******* 2021-02-12T17:51:31.250 INFO:teuthology.task.ansible.out: TASK [common : Check firewalld status] ***************************************** fatal: [smithi107.front.sepia.ceph.com]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": true} ...ignoring 2021-02-12T17:51:31.283 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:31 +0000 (0:00:00.279) 0:01:44.674 ******* 2021-02-12T17:51:31.309 INFO:teuthology.task.ansible.out: TASK [common : Open nrpe port if firewalld enabled] **************************** 2021-02-12T17:51:31.310 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:51:31.341 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:31 +0000 (0:00:00.057) 0:01:44.732 ******* 2021-02-12T17:51:31.778 INFO:teuthology.task.ansible.out: TASK [common : Upload nagios nrpe config.] ************************************* 2021-02-12T17:51:31.779 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "aa9150d4582f919d7a8504c01c8d86f05e3eeb5d", "dest": "/etc/nagios/nrpe.cfg", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/nagios/nrpe.cfg", "secontext": "system_u:object_r:nrpe_etc_t:s0", "size": 1184, "state": "file", "uid": 0} 2021-02-12T17:51:31.812 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:31 +0000 (0:00:00.471) 0:01:45.204 ******* 2021-02-12T17:51:32.518 INFO:teuthology.task.ansible.out: TASK [common : Make sure nagios nrpe service is running.] ********************** 2021-02-12T17:51:32.520 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "nrpe", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-02-12 17:49:06 UTC", "ActiveEnterTimestampMonotonic": "13217588", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "system.slice time-sync.target sysinit.target -.mount tmp.mount basic.target network.target local-fs.target var-run.mount nss-lookup.target systemd-tmpfiles-setup.service systemd-journald.socket", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-02-12 17:49:06 UTC", "AssertTimestampMonotonic": "13216535", "Before": "xdm.service shutdown.target plymouth-quit.service multi-user.target getty@tty1.service", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-02-12 17:49:06 UTC", "ConditionTimestampMonotonic": "13216535", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target nrpe.socket", "Co 2021-02-12T17:51:32.521 INFO:teuthology.task.ansible.out:ntrolGroup": "/system.slice/nrpe.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "Nagios Remote Plugin Executor", "DevicePolicy": "auto", "Documentation": "http://www.nagios.org/documentation", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "935", "ExecMainStartTimestamp": "Fri 2021-02-12 17:49:06 UTC", "ExecMainStartTimestampMonotonic": "13217546", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/nrpe ; argv[]=/usr/sbin/nrpe -c /etc/nagios/nrpe.cfg -f ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStopPost": "{ path=/bin/rm ; argv[]=/bin/rm -f /var/run/nrpe/nrpe.pid ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/nrpe.service", "GID": "984", "Group": "nrpe", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "nrpe.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-02-12 17:49:06 UTC", "InactiveExitTimestampMonotonic": "13217588", "InvocationID": "f41c2fed3e1d40fb8961ff604d3d112b", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", " 2021-02-12T17:51:32.522 INFO:teuthology.task.ansible.out:LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127260", "LimitNPROCSoft": "127260", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127260", "LimitSIGPENDINGSoft": "127260", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "935", "MemoryAccounting": "yes", "MemoryCurrent": "1642496", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "nrpe.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "-500", "OnFailureJobMode": "replace", "PIDFile": "/var/run/nrpe/nrpe.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "system.slice -.mount sysinit.target", "RequiresMountsFor": "/var/tmp /run/nrpe", 2021-02-12T17:51:32.524 INFO:teuthology.task.ansible.out:"Restart": "on-abort", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectory": "nrpe", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-02-12 17:49:06 UTC", "StateChangeTimestampMonotonic": "13217588", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203616", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min", "TimerSlackNSec": "50000", "Transient": "no", "Type": "simple", "UID": "988", "UMask": "0022", "UnitFilePreset": "disabled", "UnitFileState": "enabled", "User": "nrpe", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-02-12 17:49:06 UTC", "WatchdogTimestampMonotonic": "13217587", "WatchdogUSec": "0"}} 2021-02-12T17:51:32.553 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:32 +0000 (0:00:00.740) 0:01:45.944 ******* 2021-02-12T17:51:32.748 INFO:teuthology.task.ansible.out: TASK [common : Get SELinux status] ********************************************* changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["getenforce"], "delta": "0:00:00.001771", "end": "2021-02-12 17:51:32.729390", "rc": 0, "start": "2021-02-12 17:51:32.727619", "stderr": "", "stderr_lines": [], "stdout": "Permissive", "stdout_lines": ["Permissive"]} 2021-02-12T17:51:32.781 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:32 +0000 (0:00:00.227) 0:01:46.172 ******* 2021-02-12T17:51:35.495 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Install SELinux dependencies] **************************** 2021-02-12T17:51:35.496 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: python3-libsemanage", "Installed: python3-policycoreutils", "Installed: python3-libsemanage-2.9-3.el8.x86_64", "Installed: libsemanage-2.9-3.el8.x86_64", "Removed: python3-libsemanage-2.9-2.el8.x86_64", "Removed: libsemanage-2.9-2.el8.x86_64"]} 2021-02-12T17:51:35.527 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:51:35 +0000 (0:00:02.746) 0:01:48.918 ******* 2021-02-12T17:52:13.912 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Ensure SELinux policy is up to date] ********************* 2021-02-12T17:52:13.913 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: selinux-policy-targeted", "Installed: selinux-policy-targeted-3.14.3-54.el8_3.2.noarch", "Installed: selinux-policy-3.14.3-54.el8_3.2.noarch", "Removed: selinux-policy-3.14.3-41.el8_2.8.noarch", "Removed: selinux-policy-targeted-3.14.3-41.el8_2.8.noarch"]} 2021-02-12T17:52:13.946 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:13 +0000 (0:00:38.418) 0:02:27.337 ******* 2021-02-12T17:52:14.620 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Set SELinux boolean nagios_run_sudo true] **************** 2021-02-12T17:52:14.622 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "name": "nagios_run_sudo", "persistent": true, "state": true} 2021-02-12T17:52:14.652 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:14 +0000 (0:00:00.706) 0:02:28.043 ******* 2021-02-12T17:52:31.444 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove SELinux policy package] *************************** 2021-02-12T17:52:31.446 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-r", "nrpe"], "delta": "0:00:16.607032", "end": "2021-02-12 17:52:31.425258", "failed_when_result": false, "rc": 0, "start": "2021-02-12 17:52:14.818226", "stderr": "", "stderr_lines": [], "stdout": "libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority).", "stdout_lines": ["libsemanage.semanage_direct_remove_key: Removing last nrpe module (no other nrpe module exists at another priority)."]} 2021-02-12T17:52:31.477 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:31 +0000 (0:00:16.824) 0:02:44.868 ******* 2021-02-12T17:52:31.926 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Copy SELinux type enforcement file] ********************** 2021-02-12T17:52:31.927 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "99d9b2ea41fb2d6613898129718064833d2de428", "dest": "/tmp/nrpe.te", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/tmp/nrpe.te", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 934, "state": "file", "uid": 0} 2021-02-12T17:52:31.958 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:31 +0000 (0:00:00.481) 0:02:45.349 ******* 2021-02-12T17:52:32.160 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Compile SELinux module file] ***************************** 2021-02-12T17:52:32.161 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["checkmodule", "-M", "-m", "-o", "/tmp/nrpe.mod", "/tmp/nrpe.te"], "delta": "0:00:00.013458", "end": "2021-02-12 17:52:32.138172", "rc": 0, "start": "2021-02-12 17:52:32.124714", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:52:32.191 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:32 +0000 (0:00:00.233) 0:02:45.582 ******* 2021-02-12T17:52:32.382 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Build SELinux policy package] **************************** 2021-02-12T17:52:32.384 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule_package", "-o", "/tmp/nrpe.pp", "-m", "/tmp/nrpe.mod"], "delta": "0:00:00.007440", "end": "2021-02-12 17:52:32.363439", "rc": 0, "start": "2021-02-12 17:52:32.355999", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:52:32.416 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:32 +0000 (0:00:00.224) 0:02:45.807 ******* 2021-02-12T17:52:49.020 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Load SELinux policy package] ***************************** 2021-02-12T17:52:49.021 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["semodule", "-i", "/tmp/nrpe.pp"], "delta": "0:00:16.415981", "end": "2021-02-12 17:52:48.997798", "rc": 0, "start": "2021-02-12 17:52:32.581817", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:52:49.053 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:49 +0000 (0:00:16.636) 0:03:02.444 ******* 2021-02-12T17:52:49.244 INFO:teuthology.task.ansible.out: TASK [common : nrpe - Remove temporary files] ********************************** 2021-02-12T17:52:49.245 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "path": "/tmp/nrpe.*", "state": "absent"} 2021-02-12T17:52:49.277 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:49 +0000 (0:00:00.224) 0:03:02.668 ******* 2021-02-12T17:52:49.302 INFO:teuthology.task.ansible.out: TASK [testnode : Include package type specific vars.] ************************** 2021-02-12T17:52:49.304 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"ceph_dependency_packages_to_remove": ["boost-random", "boost-program-options", "leveldb", "xmlstarlet", "python-jinja2", "python-ceph", "python-flask", "python-requests", "boost-random", "python-urllib3", "python-babel", "hdparm", "python-markupsafe", "python-werkzeug", "python-itsdangerous"], "ceph_packages_to_remove": ["ceph", "ceph-base", "ceph-selinux", "ceph-common", "ceph-debuginfo", "ceph-release", "libcephfs1", "ceph-radosgw", "python-ceph", "python-rados", "python-rbd", "python-cephfs", "librbd1", "librados2", "mod_fastcgi"], "ntp_service_name": "ntpd", "packages_to_remove": ["device-mapper-multipath"], "pip_packages_to_install": ["remoto>=0.0.35"], "ssh_service_name": "sshd"}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/dnf_systems.yml"], "changed": false} 2021-02-12T17:52:49.337 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:49 +0000 (0:00:00.059) 0:03:02.728 ******* 2021-02-12T17:52:49.373 INFO:teuthology.task.ansible.out: TASK [testnode : Including distro specific variables.] ************************* 2021-02-12T17:52:49.374 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-12T17:52:49.406 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:49 +0000 (0:00:00.069) 0:03:02.798 ******* 2021-02-12T17:52:49.445 INFO:teuthology.task.ansible.out: TASK [testnode : Including major version specific variables.] ****************** 2021-02-12T17:52:49.447 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml) => {"ansible_facts": {"common_yum_repos": {"lab-extras": {"baseurl": "http://{{ mirror_host }}/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "epel_packages": [], "nfs_service": "nfs-server", "ntp_service_name": "chronyd", "packages": ["dnf-utils", "sysstat", "libedit", "boost-thread", "xfsprogs", "gdisk", "parted", "libgcrypt", "fuse-libs", "openssl", "libuuid", "podman", "podman-docker", "attr", "ant", "lsof", "gettext", "bc", "xfsdump", "blktrace", "usbredir", "libev-devel", "valgrind", "nfs-utils", "ncurses-devel", "gcc", "git", "python3-nose", "python3-virtualenv", "genisoimage", "qemu-img", "qemu-kvm-core", "qemu-kvm-block-rbd", "libacl-devel", "dbench", "autoconf", "gdb", "iozone"], "yum_mirrorlists": ["CentOS-AppStream-mirrorlist", "CentOS-Base-mirrorlist", "CentOS-Extras-mirrorlist"], "yum_repos": {"CentOS-AppStream": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}, "CentOS-Base": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}, "CentOS-Extras": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/centos_8.yml"} 2021-02-12T17:52:49.479 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:49 +0000 (0:00:00.072) 0:03:02.871 ******* 2021-02-12T17:52:49.516 INFO:teuthology.task.ansible.out: TASK [testnode : Including version specific variables.] ************************ 2021-02-12T17:52:49.517 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml) => {"ansible_facts": {}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/testnode/vars/empty.yml"} 2021-02-12T17:52:49.550 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:49 +0000 (0:00:00.070) 0:03:02.941 ******* 2021-02-12T17:52:49.731 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the sudo group exists.] ******************************** 2021-02-12T17:52:49.733 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 1000, "name": "sudo", "state": "present", "system": false} 2021-02-12T17:52:49.763 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:49 +0000 (0:00:00.212) 0:03:03.154 ******* 2021-02-12T17:52:49.953 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the teuthology_user group exists.] ********************* 2021-02-12T17:52:49.954 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "name": "ubuntu", "state": "present", "system": false} 2021-02-12T17:52:49.987 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:49 +0000 (0:00:00.223) 0:03:03.378 ******* 2021-02-12T17:52:50.013 INFO:teuthology.task.ansible.out: TASK [testnode : Create the teuthology user.] ********************************** 2021-02-12T17:52:50.014 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:52:50.046 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:50 +0000 (0:00:00.059) 0:03:03.438 ******* 2021-02-12T17:52:50.072 INFO:teuthology.task.ansible.out: TASK [testnode : Delete the teuthology users password.] ************************ 2021-02-12T17:52:50.073 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:52:50.104 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:50 +0000 (0:00:00.057) 0:03:03.495 ******* 2021-02-12T17:52:50.305 INFO:teuthology.task.ansible.out: TASK [testnode : Add a user for xfstests to test user quotas.] ***************** 2021-02-12T17:52:50.306 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"append": false, "changed": false, "comment": "", "group": 10101, "home": "/home/fsgqa", "move_home": false, "name": "fsgqa", "shell": "/bin/bash", "state": "present", "uid": 10101} 2021-02-12T17:52:50.339 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:50 +0000 (0:00:00.235) 0:03:03.730 ******* 2021-02-12T17:52:50.850 INFO:teuthology.task.ansible.out: TASK [testnode : Set a high max open files limit for the teuthology user.] ***** 2021-02-12T17:52:51.068 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "dc140f4bb26594b8d5e0753bab8d93c0cced7772", "dest": "/etc/security/limits.d/ubuntu.conf", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/security/limits.d/ubuntu.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 112, "state": "file", "uid": 0} Friday 12 February 2021 17:52:50 +0000 (0:00:00.544) 0:03:04.275 ******* TASK [testnode : Set hostname var] ********************************************* ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"hostname": "smithi107"}, "changed": false} Friday 12 February 2021 17:52:50 +0000 (0:00:00.059) 0:03:04.335 ******* 2021-02-12T17:52:51.936 INFO:teuthology.task.ansible.out: TASK [testnode : Set the system's hostname] ************************************ 2021-02-12T17:52:51.938 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"ansible_domain": "front.sepia.ceph.com", "ansible_fqdn": "smithi107.front.sepia.ceph.com", "ansible_hostname": "smithi107", "ansible_nodename": "smithi107"}, "changed": true, "name": "smithi107"} 2021-02-12T17:52:51.968 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:51 +0000 (0:00:01.024) 0:03:05.359 ******* 2021-02-12T17:52:52.418 INFO:teuthology.task.ansible.out: TASK [testnode : Upload distro major version specific sshd_config] ************* 2021-02-12T17:52:52.419 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "a5e550b2aa74f6355e926d70d73a50ad9ab4b4be", "dest": "/etc/ssh/sshd_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/sshd_config", "secontext": "system_u:object_r:etc_t:s0", "size": 1043, "state": "file", "uid": 0} 2021-02-12T17:52:52.451 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:52 +0000 (0:00:00.482) 0:03:05.842 ******* 2021-02-12T17:52:52.889 INFO:teuthology.task.ansible.out: TASK [testnode : Upload ssh_config] ******************************************** 2021-02-12T17:52:52.890 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "58d78b02605c8368f1913de12e145f4ad930fafc", "dest": "/etc/ssh/ssh_config", "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/etc/ssh/ssh_config", "secontext": "system_u:object_r:etc_t:s0", "size": 570, "state": "file", "uid": 0} 2021-02-12T17:52:52.922 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:52 +0000 (0:00:00.471) 0:03:06.313 ******* 2021-02-12T17:52:53.560 INFO:teuthology.task.ansible.out: TASK [testnode : Add ssh pubkeys] ********************************************** 2021-02-12T17:52:53.561 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "comment": null, "exclusive": false, "follow": false, "key": "https://raw.githubusercontent.com/ceph/keys/autogenerated/ssh/@all.pub", "key_options": null, "keyfile": "/home/ubuntu/.ssh/authorized_keys", "manage_dir": true, "path": null, "state": "present", "user": "ubuntu", "validate_certs": true} 2021-02-12T17:52:53.593 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:53 +0000 (0:00:00.671) 0:03:06.985 ******* 2021-02-12T17:52:54.047 INFO:teuthology.task.ansible.out: TASK [testnode : Create remote.conf] ******************************************* 2021-02-12T17:52:54.049 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "68d0b8ee2f88a77895f257aa69f32a42e47f455b", "dest": "/etc/security/limits.d/remote.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.d/remote.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 116, "state": "file", "uid": 0} 2021-02-12T17:52:54.081 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:54 +0000 (0:00:00.487) 0:03:07.472 ******* 2021-02-12T17:52:54.273 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-12T17:52:54.274 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "dest": "/etc/fuse.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-12T17:52:54.308 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:54 +0000 (0:00:00.226) 0:03:07.699 ******* 2021-02-12T17:52:54.500 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-12T17:52:54.501 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 36, "name": "kvm", "state": "present", "system": false} 2021-02-12T17:52:54.537 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:54 +0000 (0:00:00.229) 0:03:07.929 ******* 2021-02-12T17:52:54.749 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-12T17:52:54.750 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"append": true, "changed": false, "comment": "", "group": 1236, "groups": "kvm,disk", "home": "/home/ubuntu", "move_home": false, "name": "ubuntu", "shell": "/bin/bash", "state": "present", "uid": 1000} 2021-02-12T17:52:54.783 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:54 +0000 (0:00:00.246) 0:03:08.175 ******* 2021-02-12T17:52:55.233 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-12T17:52:55.234 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "be3223c8b0917bde02657a5f9ddd95277cc3f812", "dest": "/etc/sudoers", "gid": 0, "group": "root", "mode": "0440", "owner": "root", "path": "/etc/sudoers", "secontext": "system_u:object_r:etc_t:s0", "size": 1808, "state": "file", "uid": 0} 2021-02-12T17:52:55.266 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:55 +0000 (0:00:00.482) 0:03:08.658 ******* 2021-02-12T17:52:55.726 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-12T17:52:55.728 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "12a2e17405b9e00ea2061e25929905d08e5fe19f", "dest": "/etc/security/limits.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/security/limits.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 854, "state": "file", "uid": 0} 2021-02-12T17:52:55.759 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:55 +0000 (0:00:00.492) 0:03:09.150 ******* 2021-02-12T17:52:57.976 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-12T17:52:57.977 INFO:teuthology.task.ansible.out:[WARNING]: Consider using the yum, dnf or zypper module rather than running 'rpm'. If you need to use command because yum, dnf or zypper is insufficient you can add 'warn: false' to this command task or set 'command_warnings=False' in ansible.cfg to get rid of this message. changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["rpm", "--rebuilddb"], "delta": "0:00:02.027846", "end": "2021-02-12 17:52:57.951371", "rc": 0, "start": "2021-02-12 17:52:55.923525", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:52:58.009 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:58 +0000 (0:00:02.250) 0:03:11.400 ******* 2021-02-12T17:52:58.480 INFO:teuthology.task.ansible.out: TASK [testnode : Configure local mirrorlists] ********************************** 2021-02-12T17:52:58.481 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=CentOS-AppStream-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "aa5ad73629677ef17172417e50d2cdadccb7a3f2", "dest": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-AppStream-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 397, "state": "file", "uid": 0} 2021-02-12T17:52:58.904 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=CentOS-Base-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "4ed3c0272e6c4ec99169cd68f0b43d40bdd4b296", "dest": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Base-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 380, "state": "file", "uid": 0} 2021-02-12T17:52:59.316 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=CentOS-Extras-mirrorlist) => {"ansible_loop_var": "item", "changed": false, "checksum": "a862f43d3b3c7b8b592aa7dce23a6d013ab20bc3", "dest": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "gid": 0, "group": "root", "item": "CentOS-Extras-mirrorlist", "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras-mirrorlist", "secontext": "system_u:object_r:system_conf_t:s0", "size": 382, "state": "file", "uid": 0} 2021-02-12T17:52:59.350 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:59 +0000 (0:00:01.340) 0:03:12.741 ******* 2021-02-12T17:52:59.808 INFO:teuthology.task.ansible.out: TASK [testnode : Configure common additional repos in /etc/yum.repos.d/] ******* 2021-02-12T17:52:59.810 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'lab-extras', 'value': {'name': 'lab-extras', 'baseurl': 'http://apt-mirror.front.sepia.ceph.com/lab-extras/8/', 'enabled': 1, 'gpgcheck': 0}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "a0e2ac0628a264e14b96b1c57627a3aa344a6c52", "dest": "/etc/yum.repos.d/lab-extras.repo", "gid": 0, "group": "root", "item": {"key": "lab-extras", "value": {"baseurl": "http://apt-mirror.front.sepia.ceph.com/lab-extras/8/", "enabled": 1, "gpgcheck": 0, "name": "lab-extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/lab-extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 203, "state": "file", "uid": 0} 2021-02-12T17:52:59.842 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:52:59 +0000 (0:00:00.492) 0:03:13.233 ******* 2021-02-12T17:53:00.302 INFO:teuthology.task.ansible.out: TASK [testnode : Configure version specific repos in /etc/yum.repos.d/] ******** 2021-02-12T17:53:00.304 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'CentOS-AppStream', 'value': {'name': 'CentOS-$releasever - AppStream', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "9b147c879e95cd23f3bbde027da102febcaad1a8", "dest": "/etc/yum.repos.d/CentOS-AppStream.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-AppStream", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-AppStream-mirrorlist", "name": "CentOS-$releasever - AppStream"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-AppStream.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 289, "state": "file", "uid": 0} 2021-02-12T17:53:00.728 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'CentOS-Base', 'value': {'name': 'CentOS-$releasever - Base', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Base-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "77e4590ed0ecec056fa44f4e65af60e110822b85", "dest": "/etc/yum.repos.d/CentOS-Base.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Base", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Base-mirrorlist", "name": "CentOS-$releasever - Base"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Base.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 274, "state": "file", "uid": 0} 2021-02-12T17:53:01.154 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'key': 'CentOS-Extras', 'value': {'name': 'CentOS-$releasever - Extras', 'mirrorlist': 'file:///etc/yum.repos.d/CentOS-Extras-mirrorlist', 'gpgcheck': 1, 'enabled': 1, 'gpgkey': 'file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial'}}) => {"ansible_loop_var": "item", "changed": false, "checksum": "2856826614c7f1cb9409228f6c41319a023ca754", "dest": "/etc/yum.repos.d/CentOS-Extras.repo", "gid": 0, "group": "root", "item": {"key": "CentOS-Extras", "value": {"enabled": 1, "gpgcheck": 1, "gpgkey": "file:///etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial", "mirrorlist": "file:///etc/yum.repos.d/CentOS-Extras-mirrorlist", "name": "CentOS-$releasever - Extras"}}, "mode": "0644", "owner": "root", "path": "/etc/yum.repos.d/CentOS-Extras.repo", "secontext": "system_u:object_r:system_conf_t:s0", "size": 280, "state": "file", "uid": 0} 2021-02-12T17:53:01.201 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:01.358) 0:03:14.592 ******* 2021-02-12T17:53:01.227 INFO:teuthology.task.ansible.out: TASK [testnode : Enable copr repos] ******************************************** 2021-02-12T17:53:01.260 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.059) 0:03:14.651 ******* 2021-02-12T17:53:01.286 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG] ************************************************** 2021-02-12T17:53:01.288 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:01.320 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.059) 0:03:14.711 ******* 2021-02-12T17:53:01.357 INFO:teuthology.task.ansible.out: TASK [testnode : Install GPG keys] ********************************************* 2021-02-12T17:53:01.359 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=https://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "https://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} 2021-02-12T17:53:01.391 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.071) 0:03:14.782 ******* 2021-02-12T17:53:01.418 INFO:teuthology.task.ansible.out: TASK [testnode : Clean yum cache] ********************************************** 2021-02-12T17:53:01.419 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:01.450 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.059) 0:03:14.842 ******* 2021-02-12T17:53:01.477 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-12T17:53:01.478 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"skip_packaging": false}, "changed": false} 2021-02-12T17:53:01.510 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.059) 0:03:14.901 ******* 2021-02-12T17:53:01.545 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure yum_utils is present.] ********************************* 2021-02-12T17:53:01.547 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:01.578 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.068) 0:03:14.970 ******* 2021-02-12T17:53:01.605 INFO:teuthology.task.ansible.out: TASK [testnode : Removing saved yum transactions] ****************************** 2021-02-12T17:53:01.606 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:01.638 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.059) 0:03:15.030 ******* 2021-02-12T17:53:01.839 INFO:teuthology.task.ansible.out: TASK [testnode : Check if ceph-debuginfo is installed] ************************* 2021-02-12T17:53:01.840 INFO:teuthology.task.ansible.out:fatal: [smithi107.front.sepia.ceph.com]: FAILED! => {"changed": false, "cmd": ["rpm", "-q", "ceph-debuginfo"], "delta": "0:00:00.007094", "end": "2021-02-12 17:53:01.818214", "msg": "non-zero return code", "rc": 1, "start": "2021-02-12 17:53:01.811120", "stderr": "", "stderr_lines": [], "stdout": "package ceph-debuginfo is not installed", "stdout_lines": ["package ceph-debuginfo is not installed"]} ...ignoring 2021-02-12T17:53:01.873 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.234) 0:03:15.264 ******* 2021-02-12T17:53:01.899 INFO:teuthology.task.ansible.out: TASK [testnode : Work around https://bugzilla.redhat.com/show_bug.cgi?id=1234967] *** 2021-02-12T17:53:01.900 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:01.933 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:01 +0000 (0:00:00.060) 0:03:15.324 ******* 2021-02-12T17:53:04.468 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-12T17:53:04.470 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: librbd1-1:12.2.7-9.el8.x86_64", "Removed: qemu-kvm-block-rbd-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: librados2-1:12.2.7-9.el8.x86_64"]} 2021-02-12T17:53:04.502 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:04 +0000 (0:00:02.568) 0:03:17.893 ******* 2021-02-12T17:53:06.612 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-12T17:53:06.613 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Removed: boost-program-options-1.66.0-6.el8.x86_64", "Removed: boost-random-1.66.0-6.el8.x86_64"]} 2021-02-12T17:53:06.646 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:06 +0000 (0:00:02.144) 0:03:20.038 ******* 2021-02-12T17:53:51.988 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-12T17:53:51.989 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: dnf-utils", "Installed: sysstat", "Installed: libedit", "Installed: boost-thread", "Installed: xfsprogs", "Installed: gdisk", "Installed: parted", "Installed: libgcrypt", "Installed: fuse-libs", "Installed: openssl", "Installed: libuuid", "Installed: podman", "Installed: podman-docker", "Installed: attr", "Installed: ant", "Installed: lsof", "Installed: gettext", "Installed: bc", "Installed: xfsdump", "Installed: blktrace", "Installed: usbredir", "Installed: libev-devel", "Installed: valgrind", "Installed: nfs-utils", "Installed: ncurses-devel", "Installed: gcc", "Installed: git", "Installed: python3-nose", "Installed: python3-virtualenv", "Installed: genisoimage", "Installed: qemu-img", "Installed: qemu-kvm-core", "Installed: qemu-kvm-block-rbd", "Installed: libacl-devel", "Installed: dbench", "Installed: autoconf", "Installed: gdb", "Installed: iozone", "Installed: util-linux-2.32.1-24.el8.x86_64", "Installed: util-linux-user-2.32.1-24.el8.x86_64", "Installed: python3-libdnf-0.48.0-5.el8.x86_64", "Installed: libdnf-0.48.0-5.el8.x86_64", "Installed: python3-librepo-1.12.0-2.el8.x86_64", "Installed: boost-chrono-1.66.0-10.el8.x86_64", "Installed: xfsprogs-5.0.0-4.el8.x86_64", "Installed: libfdisk-2.32.1-24.el8.x86_64", "Installed: librados2-1:12.2.7-9.el8.x86_64", "Installed: gdb-8.2-12.el8.x86_64", "Installed: librbd1-1:12.2.7-9.el8.x86_64", "Installed: nfs-utils-1:2.3.3-35.el8.x86_64", "Installed: gcc-8.3.1-5.1.el8.x86_64", "Installed: gdb-headless-8.2-12.el8.x86_64", "Installed: libuuid-2.32.1-24.el8.x86_64", "Installed: boost-program-options-1.66.0-10.el8.x86_64", "Installed: gcc-gdb-plugin-8.3.1-5.1.el8.x86_64", "Installed: boost-random-1.66.0-10.el8.x86_64", "Installed: python3-virtualenv-15.1.0-19.module_el8.3.0+562+e162826a.noarch", "Installed: libgcc-8.3.1-5.1.el8.x86_64", "Installed: libgcrypt-1.8.5-4.el8.x86_64", "Installed: yum-utils-4.0.17-5.el8.no 2021-02-12T17:53:51.991 INFO:teuthology.task.ansible.out:arch", "Installed: libgomp-8.3.1-5.1.el8.x86_64", "Installed: yum-4.2.23-4.el8.noarch", "Installed: perl-Git-2.27.0-1.el8.noarch", "Installed: boost-system-1.66.0-10.el8.x86_64", "Installed: boost-thread-1.66.0-10.el8.x86_64", "Installed: librepo-1.12.0-2.el8.x86_64", "Installed: podman-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64", "Installed: podman-catatonit-2.0.5-5.module_el8.3.0+512+b3b58dca.x86_64", "Installed: podman-docker-2.0.5-5.module_el8.3.0+512+b3b58dca.noarch", "Installed: dnf-4.2.23-4.el8.noarch", "Installed: dnf-data-4.2.23-4.el8.noarch", "Installed: dnf-plugins-core-4.0.17-5.el8.noarch", "Installed: python3-nose-1.3.7-30.module_el8.3.0+562+e162826a.noarch", "Installed: libsmartcols-2.32.1-24.el8.x86_64", "Installed: openssl-1:1.1.1g-12.el8_3.x86_64", "Installed: lsof-4.93.2-1.el8.x86_64", "Installed: valgrind-1:3.16.0-2.el8.x86_64", "Installed: libblkid-2.32.1-24.el8.x86_64", "Installed: git-2.27.0-1.el8.x86_64", "Installed: openssl-libs-1:1.1.1g-12.el8_3.x86_64", "Installed: git-core-2.27.0-1.el8.x86_64", "Installed: git-core-doc-2.27.0-1.el8.noarch", "Installed: qemu-img-15:4.2.0-34.module_el8.3.0+613+9ec9f184.1.x86_64", "Installed: qemu-kvm-block-rbd-15:4.2.0-34.module_el8.3.0+613+9ec9f184.1.x86_64", "Installed: qemu-kvm-common-15:4.2.0-34.module_el8.3.0+613+9ec9f184.1.x86_64", "Installed: qemu-kvm-core-15:4.2.0-34.module_el8.3.0+613+9ec9f184.1.x86_64", "Installed: python3-hawkey-0.48.0-5.el8.x86_64", "Installed: cpp-8.3.1-5.1.el8.x86_64", "Installed: libmodulemd-2.9.4-2.el8.x86_64", "Installed: libmount-2.32.1-24.el8.x86_64", "Installed: python3-dnf-4.2.23-4.el8.noarch", "Installed: python3-dnf-plugins-core-4.0.17-5.el8.noarch", "Installed: sysstat-11.7.3-5.el8.x86_64", "Installed: seavgabios-bin-1.13.0-2.module_el8.3.0+555+a55c8938.noarch", "Removed: python3-virtualenv-15.1.0-19.module_el8.1.0+245+c39af44f.noarch", "Removed: openssl-1:1.1.1c-15.el8.x86_64", "Removed: qemu-img-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: qemu-kvm- 2021-02-12T17:53:51.992 INFO:teuthology.task.ansible.out:common-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: qemu-kvm-core-15:2.12.0-88.module_el8.1.0+297+df420408.3.x86_64", "Removed: openssl-libs-1:1.1.1c-15.el8.x86_64", "Removed: librepo-1.11.0-2.el8.x86_64", "Removed: libsmartcols-2.32.1-22.el8.x86_64", "Removed: gcc-8.3.1-4.5.el8.x86_64", "Removed: gcc-gdb-plugin-8.3.1-4.5.el8.x86_64", "Removed: gdb-8.2-6.el8.x86_64", "Removed: gdb-headless-8.2-6.el8.x86_64", "Removed: seavgabios-bin-1.11.1-4.module_el8.1.0+248+298dec18.noarch", "Removed: boost-chrono-1.66.0-6.el8.x86_64", "Removed: git-2.18.2-2.el8_1.x86_64", "Removed: git-core-2.18.2-2.el8_1.x86_64", "Removed: git-core-doc-2.18.2-2.el8_1.noarch", "Removed: boost-system-1.66.0-6.el8.x86_64", "Removed: boost-thread-1.66.0-6.el8.x86_64", "Removed: libuuid-2.32.1-22.el8.x86_64", "Removed: podman-1.6.4-4.module_el8.1.0+298+41f9343a.x86_64", "Removed: podman-docker-1.6.4-4.module_el8.1.0+298+41f9343a.noarch", "Removed: podman-manpages-1.6.4-4.module_el8.1.0+298+41f9343a.noarch", "Removed: libblkid-2.32.1-22.el8.x86_64", "Removed: libdnf-0.35.1-9.el8_1.x86_64", "Removed: perl-Git-2.18.2-2.el8_1.noarch", "Removed: lsof-4.91-2.el8.x86_64", "Removed: cpp-8.3.1-4.5.el8.x86_64", "Removed: sysstat-11.7.3-2.el8.x86_64", "Removed: libfdisk-2.32.1-22.el8.x86_64", "Removed: python3-dnf-4.2.7-7.el8_1.noarch", "Removed: python3-dnf-plugins-core-4.0.8-3.el8.noarch", "Removed: libgcc-8.3.1-5.el8.0.2.x86_64", "Removed: libgcrypt-1.8.3-4.el8.x86_64", "Removed: python3-hawkey-0.35.1-9.el8_1.x86_64", "Removed: libgomp-8.3.1-4.5.el8.x86_64", "Removed: python3-libdnf-0.35.1-9.el8_1.x86_64", "Removed: python3-librepo-1.11.0-2.el8.x86_64", "Removed: util-linux-2.32.1-22.el8.x86_64", "Removed: util-linux-user-2.32.1-22.el8.x86_64", "Removed: valgrind-1:3.15.0-9.el8.x86_64", "Removed: python3-nose-1.3.7-30.module_el8.1.0+245+c39af44f.noarch", "Removed: dnf-4.2.7-7.el8_1.noarch", "Removed: dnf-data-4.2.7-7.el8_1.noarch", "Removed: libmount-2.32.1-22.el8.x86_64", "Removed: dnf-plu 2021-02-12T17:53:51.993 INFO:teuthology.task.ansible.out:gins-core-4.0.8-3.el8.noarch", "Removed: nfs-utils-1:2.3.3-26.el8.x86_64", "Removed: xfsprogs-5.0.0-2.el8.x86_64", "Removed: yum-4.2.7-7.el8_1.noarch", "Removed: yum-utils-4.0.8-3.el8.noarch"]} 2021-02-12T17:53:52.047 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:52 +0000 (0:00:45.400) 0:04:05.438 ******* 2021-02-12T17:53:52.086 INFO:teuthology.task.ansible.out: TASK [testnode : Install epel packages] **************************************** 2021-02-12T17:53:52.087 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:52.121 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:52 +0000 (0:00:00.073) 0:04:05.512 ******* 2021-02-12T17:53:53.184 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-12T17:53:53.186 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": []} 2021-02-12T17:53:53.218 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:53 +0000 (0:00:01.096) 0:04:06.609 ******* 2021-02-12T17:53:53.245 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-12T17:53:53.246 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:53.279 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:53 +0000 (0:00:00.060) 0:04:06.670 ******* 2021-02-12T17:53:53.575 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable firewalld] *********************************** 2021-02-12T17:53:53.576 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "enabled": false, "name": "firewalld", "state": "stopped", "status": {"ActiveEnterTimestampMonotonic": "0", "ActiveExitTimestampMonotonic": "0", "ActiveState": "inactive", "After": "sysinit.target system.slice basic.target dbus.socket dbus.service polkit.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "no", "AssertTimestampMonotonic": "0", "Before": "network-pre.target shutdown.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "BusName": "org.fedoraproject.FirewallD1", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "yes", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "no", "ConditionTimestampMonotonic": "0", "ConfigurationDirectoryMode": "0755", "Conflicts": "iptables.service shutdown.target ebtables.service ipset.service ip6tables.service", "ControlPID": "0", "DefaultDependencies": "yes", "Delegate": "no", "Description": "firewalld - dynamic firewall daemon", "DevicePolicy": "auto", "Documentation": "man:firewalld(1)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNode 2021-02-12T17:53:53.578 INFO:teuthology.task.ansible.out:s": "", "EnvironmentFiles": "/etc/sysconfig/firewalld (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "0", "ExecMainStartTimestampMonotonic": "0", "ExecMainStatus": "0", "ExecReload": "{ path=/bin/kill ; argv[]=/bin/kill -HUP $MAINPID ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStart": "{ path=/usr/sbin/firewalld ; argv[]=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/firewalld.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "firewalld.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestampMonotonic": "0", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "mixed", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127260", "LimitNPROCSoft": "127260", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "L 2021-02-12T17:53:53.579 INFO:teuthology.task.ansible.out:imitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127260", "LimitSIGPENDINGSoft": "127260", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "0", "MemoryAccounting": "yes", "MemoryCurrent": "[not set]", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "firewalld.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "no", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "no", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "no", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "sysinit.target system.slice dbus.socket", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOnly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "null", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "null", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWe 2021-02-12T17:53:53.580 INFO:teuthology.task.ansible.out:ight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestampMonotonic": "0", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "dead", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "[not set]", "TasksMax": "203616", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "dbus", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "disabled", "UtmpMode": "init", "Wants": "network-pre.target", "WatchdogTimestampMonotonic": "0", "WatchdogUSec": "0"}} 2021-02-12T17:53:53.610 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:53 +0000 (0:00:00.331) 0:04:07.001 ******* 2021-02-12T17:53:53.897 INFO:teuthology.task.ansible.out: TASK [testnode : Stop and disable iptables] ************************************ 2021-02-12T17:53:53.899 INFO:teuthology.task.ansible.out:fatal: [smithi107.front.sepia.ceph.com]: FAILED! => {"changed": false, "msg": "Could not find the requested service iptables: host"} ...ignoring 2021-02-12T17:53:53.931 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:53 +0000 (0:00:00.320) 0:04:07.322 ******* 2021-02-12T17:53:54.631 INFO:teuthology.task.ansible.out: TASK [testnode : Enable SELinux] *********************************************** 2021-02-12T17:53:54.632 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "configfile": "/etc/selinux/config", "msg": "", "policy": "targeted", "reboot_required": false, "state": "permissive"} 2021-02-12T17:53:54.665 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:54 +0000 (0:00:00.734) 0:04:08.056 ******* 2021-02-12T17:53:54.691 INFO:teuthology.task.ansible.out: TASK [testnode : Install abrt] ************************************************* 2021-02-12T17:53:54.692 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:54.725 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:54 +0000 (0:00:00.059) 0:04:08.116 ******* 2021-02-12T17:53:54.751 INFO:teuthology.task.ansible.out: TASK [testnode : Enable abrt-auto-reporting] *********************************** 2021-02-12T17:53:54.752 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:54.785 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:54 +0000 (0:00:00.060) 0:04:08.176 ******* 2021-02-12T17:53:54.811 INFO:teuthology.task.ansible.out: TASK [testnode : Set OpenGPGCheck in abrt-action-save-package-data.conf] ******* 2021-02-12T17:53:54.812 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:54.845 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:54 +0000 (0:00:00.060) 0:04:08.236 ******* 2021-02-12T17:53:54.871 INFO:teuthology.task.ansible.out: TASK [testnode : Set ProcessUnpackaged in abrt-action-save-package-data.conf] *** 2021-02-12T17:53:54.873 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:54.903 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:54 +0000 (0:00:00.058) 0:04:08.295 ******* 2021-02-12T17:53:54.930 INFO:teuthology.task.ansible.out: TASK [testnode : Restart abrtd] ************************************************ 2021-02-12T17:53:54.931 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:54.962 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:54 +0000 (0:00:00.058) 0:04:08.354 ******* 2021-02-12T17:53:54.988 INFO:teuthology.task.ansible.out: TASK [testnode : Check for custom repos] *************************************** 2021-02-12T17:53:54.990 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.021 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.059) 0:04:08.413 ******* 2021-02-12T17:53:55.048 INFO:teuthology.task.ansible.out: TASK [testnode : Remove custom repos] ****************************************** 2021-02-12T17:53:55.081 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.059) 0:04:08.472 ******* 2021-02-12T17:53:55.107 INFO:teuthology.task.ansible.out: TASK [testnode : Set apt preferences] ****************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.139 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.057) 0:04:08.530 ******* 2021-02-12T17:53:55.165 INFO:teuthology.task.ansible.out: TASK [testnode : Add sources list] ********************************************* 2021-02-12T17:53:55.166 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.197 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.058) 0:04:08.589 ******* 2021-02-12T17:53:55.234 INFO:teuthology.task.ansible.out: TASK [testnode : Install apt keys] ********************************************* 2021-02-12T17:53:55.235 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/autobuild.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/autobuild.asc", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=http://download.ceph.com/keys/release.asc) => {"ansible_loop_var": "item", "changed": false, "item": "http://download.ceph.com/keys/release.asc", "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.268 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.070) 0:04:08.659 ******* 2021-02-12T17:53:55.295 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-apt] ******************************************* 2021-02-12T17:53:55.296 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.330 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.061) 0:04:08.721 ******* 2021-02-12T17:53:55.356 INFO:teuthology.task.ansible.out: TASK [testnode : Add local apt repos.] ***************************************** 2021-02-12T17:53:55.411 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.081) 0:04:08.802 ******* 2021-02-12T17:53:55.440 INFO:teuthology.task.ansible.out: TASK [testnode : Blast the apt cache.] ***************************************** 2021-02-12T17:53:55.441 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.483 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.071) 0:04:08.874 ******* 2021-02-12T17:53:55.509 INFO:teuthology.task.ansible.out: TASK [testnode : Update apt cache.] ******************************************** 2021-02-12T17:53:55.510 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.542 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.059) 0:04:08.933 ******* 2021-02-12T17:53:55.568 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure packages are not present.] ***************************** 2021-02-12T17:53:55.569 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.601 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.058) 0:04:08.992 ******* 2021-02-12T17:53:55.627 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.660 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.058) 0:04:09.051 ******* 2021-02-12T17:53:55.690 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade non aarch64 packages] ********************************* 2021-02-12T17:53:55.691 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.722 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.062) 0:04:09.114 ******* 2021-02-12T17:53:55.749 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.782 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.059) 0:04:09.173 ******* 2021-02-12T17:53:55.808 INFO:teuthology.task.ansible.out: TASK [testnode : Install non aarch64 packages] ********************************* 2021-02-12T17:53:55.809 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.840 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.058) 0:04:09.231 ******* 2021-02-12T17:53:55.866 INFO:teuthology.task.ansible.out: TASK [testnode : Use perl to add user_xattr to the root mount options in fstab.] *** 2021-02-12T17:53:55.867 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.899 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.058) 0:04:09.290 ******* 2021-02-12T17:53:55.924 INFO:teuthology.task.ansible.out: TASK [testnode : Enable xattr for this boot.] ********************************** 2021-02-12T17:53:55.925 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:55.957 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:55 +0000 (0:00:00.058) 0:04:09.349 ******* 2021-02-12T17:53:55.983 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure fuse group exists.] ************************************ 2021-02-12T17:53:55.984 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.016 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.058) 0:04:09.407 ******* 2021-02-12T17:53:56.041 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/fuse.conf.] *************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.074 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.058) 0:04:09.466 ******* 2021-02-12T17:53:56.100 INFO:teuthology.task.ansible.out: TASK [testnode : Add teuthology user to groups fuse, kvm and disk.] ************ skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.134 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.059) 0:04:09.525 ******* 2021-02-12T17:53:56.160 INFO:teuthology.task.ansible.out: TASK [testnode : Set up static IP in /etc/hosts.] ****************************** 2021-02-12T17:53:56.161 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.193 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.059) 0:04:09.585 ******* 2021-02-12T17:53:56.219 INFO:teuthology.task.ansible.out: TASK [testnode : Stop apache2] ************************************************* 2021-02-12T17:53:56.220 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.253 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.059) 0:04:09.644 ******* 2021-02-12T17:53:56.279 INFO:teuthology.task.ansible.out: TASK [testnode : Set mode on /etc/fuse.conf] *********************************** 2021-02-12T17:53:56.280 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.312 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.059) 0:04:09.703 ******* 2021-02-12T17:53:56.338 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure the group kvm exists.] ********************************* 2021-02-12T17:53:56.340 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.371 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.059) 0:04:09.763 ******* 2021-02-12T17:53:56.397 INFO:teuthology.task.ansible.out: TASK [testnode : Add the teuthology user to groups kvm,disk] ******************* 2021-02-12T17:53:56.399 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.430 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.058) 0:04:09.821 ******* 2021-02-12T17:53:56.456 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/sudoers.] ************************************** 2021-02-12T17:53:56.457 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.489 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.058) 0:04:09.880 ******* 2021-02-12T17:53:56.515 INFO:teuthology.task.ansible.out: TASK [testnode : Configure /etc/security/limits.conf] ************************** 2021-02-12T17:53:56.516 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.551 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.061) 0:04:09.942 ******* 2021-02-12T17:53:56.578 INFO:teuthology.task.ansible.out: TASK [testnode : Rebuild rpmdb] ************************************************ 2021-02-12T17:53:56.579 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.612 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.061) 0:04:10.004 ******* 2021-02-12T17:53:56.638 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph packages are not present.] ************************ 2021-02-12T17:53:56.640 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.674 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.061) 0:04:10.065 ******* 2021-02-12T17:53:56.700 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure ceph dependency packages are not present.] ************* 2021-02-12T17:53:56.702 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.736 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.061) 0:04:10.127 ******* 2021-02-12T17:53:56.763 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-12T17:53:56.764 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.797 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.061) 0:04:10.188 ******* 2021-02-12T17:53:56.823 INFO:teuthology.task.ansible.out: TASK [testnode : Remove packages] ********************************************** 2021-02-12T17:53:56.824 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.858 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.061) 0:04:10.249 ******* 2021-02-12T17:53:56.885 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages] ********************************************* 2021-02-12T17:53:56.886 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.928 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.069) 0:04:10.319 ******* 2021-02-12T17:53:56.955 INFO:teuthology.task.ansible.out: TASK [testnode : Upgrade packages] ********************************************* 2021-02-12T17:53:56.957 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:56.989 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:56 +0000 (0:00:00.061) 0:04:10.380 ******* 2021-02-12T17:53:57.016 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-12T17:53:57.017 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:57.049 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:57 +0000 (0:00:00.059) 0:04:10.440 ******* 2021-02-12T17:53:57.263 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-12T17:53:57.263 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/bin/lspci", "gid": 0, "group": "root", "mode": "0777", "owner": "root", "secontext": "unconfined_u:object_r:bin_t:s0", "size": 11, "src": "/sbin/lspci", "state": "link", "uid": 0} 2021-02-12T17:53:57.297 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:57 +0000 (0:00:00.247) 0:04:10.688 ******* 2021-02-12T17:53:57.499 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-12T17:53:57.500 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/usr/lib/ltp/testcases/bin", "secontext": "unconfined_u:object_r:lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-12T17:53:57.533 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:57 +0000 (0:00:00.236) 0:04:10.924 ******* 2021-02-12T17:53:57.727 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-12T17:53:57.728 INFO:teuthology.task.ansible.out:[WARNING]: Cannot set fs attributes on a non-existent symlink target. follow should be set to False to avoid this. ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "dest": "/usr/lib/ltp/testcases/bin/fsstress", "src": "/usr/bin/fsstress"} 2021-02-12T17:53:57.761 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:57 +0000 (0:00:00.228) 0:04:11.152 ******* 2021-02-12T17:53:57.788 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-12T17:53:57.789 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:57.822 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:57 +0000 (0:00:00.060) 0:04:11.213 ******* 2021-02-12T17:53:57.852 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:57.884 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:57 +0000 (0:00:00.062) 0:04:11.276 ******* 2021-02-12T17:53:57.910 INFO:teuthology.task.ansible.out: TASK [testnode : Include hostname in /etc/sysconfig/network] ******************* 2021-02-12T17:53:57.912 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:57.944 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:57 +0000 (0:00:00.059) 0:04:11.335 ******* 2021-02-12T17:53:57.970 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-12T17:53:57.971 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.004 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.060) 0:04:11.395 ******* 2021-02-12T17:53:58.030 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-12T17:53:58.031 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.063 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.058) 0:04:11.454 ******* 2021-02-12T17:53:58.089 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-12T17:53:58.090 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.122 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.059) 0:04:11.514 ******* 2021-02-12T17:53:58.149 INFO:teuthology.task.ansible.out: TASK [testnode : Make raid/smart scripts work.] ******************************** 2021-02-12T17:53:58.150 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.181 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.058) 0:04:11.572 ******* 2021-02-12T17:53:58.206 INFO:teuthology.task.ansible.out: TASK [testnode : Create FStest ubuntu directory.] ****************************** 2021-02-12T17:53:58.208 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.238 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.057) 0:04:11.630 ******* 2021-02-12T17:53:58.264 INFO:teuthology.task.ansible.out: TASK [testnode : Make fsstress same path as ubuntu.] *************************** 2021-02-12T17:53:58.266 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.297 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.058) 0:04:11.688 ******* 2021-02-12T17:53:58.323 INFO:teuthology.task.ansible.out: TASK [testnode : Set grub config.] ********************************************* 2021-02-12T17:53:58.324 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.358 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.060) 0:04:11.749 ******* 2021-02-12T17:53:58.384 INFO:teuthology.task.ansible.out: TASK [testnode : Remove /etc/ceph] ********************************************* 2021-02-12T17:53:58.386 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.417 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.059) 0:04:11.808 ******* 2021-02-12T17:53:58.443 INFO:teuthology.task.ansible.out: TASK [testnode : Upload a dummy nfs export so that the nfs kernel server starts.] *** 2021-02-12T17:53:58.445 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.476 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.058) 0:04:11.867 ******* 2021-02-12T17:53:58.502 INFO:teuthology.task.ansible.out: TASK [testnode : Enable nfs-server on rhel 7.x.] ******************************* 2021-02-12T17:53:58.504 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.535 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.058) 0:04:11.926 ******* 2021-02-12T17:53:58.560 INFO:teuthology.task.ansible.out: TASK [testnode : Upload weekly kernel-clean crontab.] ************************** 2021-02-12T17:53:58.562 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.593 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.057) 0:04:11.984 ******* 2021-02-12T17:53:58.618 INFO:teuthology.task.ansible.out: TASK [testnode : Upload /etc/grub.d/02_force_timeout.] ************************* 2021-02-12T17:53:58.620 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.652 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.059) 0:04:12.043 ******* 2021-02-12T17:53:58.678 INFO:teuthology.task.ansible.out: TASK [testnode : Enable kernel modules to load at boot time.] ****************** 2021-02-12T17:53:58.679 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.714 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.062) 0:04:12.106 ******* 2021-02-12T17:53:58.741 INFO:teuthology.task.ansible.out: TASK [testnode : Enabling auto-fsck fix to prevent boot hangup.] *************** 2021-02-12T17:53:58.742 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.774 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.059) 0:04:12.165 ******* 2021-02-12T17:53:58.800 INFO:teuthology.task.ansible.out: TASK [testnode : Work around broken wget on wheezy.] *************************** 2021-02-12T17:53:58.802 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.834 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.060) 0:04:12.226 ******* 2021-02-12T17:53:58.861 INFO:teuthology.task.ansible.out: TASK [testnode : Stop collectl] ************************************************ 2021-02-12T17:53:58.862 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.894 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.059) 0:04:12.285 ******* 2021-02-12T17:53:58.921 INFO:teuthology.task.ansible.out: TASK [testnode : Add PATH to the teuthology_user .bashrc.] ********************* 2021-02-12T17:53:58.922 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:58.953 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:58 +0000 (0:00:00.058) 0:04:12.344 ******* 2021-02-12T17:53:58.979 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if we need to edit /etc/profile.] **************** 2021-02-12T17:53:58.980 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.012 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.059) 0:04:12.404 ******* 2021-02-12T17:53:59.038 INFO:teuthology.task.ansible.out: TASK [testnode : Update /etc/profile if needed.] ******************************* 2021-02-12T17:53:59.040 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.072 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.059) 0:04:12.464 ******* 2021-02-12T17:53:59.098 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-12T17:53:59.100 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"nvme_card_present": false}, "changed": false} 2021-02-12T17:53:59.132 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.059) 0:04:12.523 ******* 2021-02-12T17:53:59.168 INFO:teuthology.task.ansible.out: TASK [testnode : Check for NVMe drive] ***************************************** 2021-02-12T17:53:59.169 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-1) => {"ansible_loop_var": "item", "changed": false, "item": "dm-1", "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.179 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=nvme0n1) => {"ansible_facts": {"nvme_card_present": true}, "ansible_loop_var": "item", "changed": false, "item": "nvme0n1"} 2021-02-12T17:53:59.190 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-4) => {"ansible_loop_var": "item", "changed": false, "item": "dm-4", "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.191 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-2) => {"ansible_loop_var": "item", "changed": false, "item": "dm-2", "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.201 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-0) => {"ansible_loop_var": "item", "changed": false, "item": "dm-0", "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.212 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=sda) => {"ansible_loop_var": "item", "changed": false, "item": "sda", "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.213 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=dm-3) => {"ansible_loop_var": "item", "changed": false, "item": "dm-3", "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.252 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.120) 0:04:12.643 ******* 2021-02-12T17:53:59.436 INFO:teuthology.task.ansible.out: TASK [testnode : Check for teuthology-lock command] **************************** 2021-02-12T17:53:59.437 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com -> localhost] => {"changed": true, "cmd": "which teuthology-lock", "delta": "0:00:00.003147", "end": "2021-02-12 17:53:59.416426", "rc": 0, "start": "2021-02-12 17:53:59.413279", "stderr": "", "stderr_lines": [], "stdout": "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/virtualenv/bin/teuthology-lock", "stdout_lines": ["/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/virtualenv/bin/teuthology-lock"]} 2021-02-12T17:53:59.472 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.220) 0:04:12.863 ******* 2021-02-12T17:53:59.498 INFO:teuthology.task.ansible.out: TASK [testnode : Mark system down if NVMe card missing] ************************ 2021-02-12T17:53:59.499 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.535 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.063) 0:04:12.927 ******* 2021-02-12T17:53:59.561 INFO:teuthology.task.ansible.out: TASK [testnode : Update description in paddles lock DB if NVMe card missing] *** 2021-02-12T17:53:59.563 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.596 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.060) 0:04:12.987 ******* 2021-02-12T17:53:59.622 INFO:teuthology.task.ansible.out: TASK [testnode : Fail rest of playbook due to missing NVMe card] *************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.658 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.061) 0:04:13.049 ******* 2021-02-12T17:53:59.684 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure apt dependencies are installed] ********************* 2021-02-12T17:53:59.685 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:53:59.717 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:53:59 +0000 (0:00:00.059) 0:04:13.108 ******* 2021-02-12T17:54:05.063 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure rpm dependencies are installed] ********************* 2021-02-12T17:54:05.064 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "msg": "", "rc": 0, "results": ["Installed: gdisk", "Installed: device-mapper", "Installed: device-mapper-event-8:1.02.171-5.el8.x86_64", "Installed: lvm2-8:2.03.09-5.el8.x86_64", "Installed: device-mapper-event-libs-8:1.02.171-5.el8.x86_64", "Installed: lvm2-libs-8:2.03.09-5.el8.x86_64", "Installed: device-mapper-libs-8:1.02.171-5.el8.x86_64", "Installed: device-mapper-8:1.02.171-5.el8.x86_64", "Removed: device-mapper-8:1.02.169-3.el8.x86_64", "Removed: device-mapper-event-8:1.02.169-3.el8.x86_64", "Removed: device-mapper-event-libs-8:1.02.169-3.el8.x86_64", "Removed: device-mapper-libs-8:1.02.169-3.el8.x86_64", "Removed: lvm2-8:2.03.08-3.el8.x86_64", "Removed: lvm2-libs-8:2.03.08-3.el8.x86_64"]} 2021-02-12T17:54:05.095 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:05 +0000 (0:00:05.378) 0:04:18.486 ******* 2021-02-12T17:54:05.131 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-12T17:54:05.132 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356162560, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137735, 'block_used': 12957373, 'inode_total': 61054976, 'inode_available': 60986354, 'inode_used': 68622, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_facts": {"root_disk": "/dev/sda"}, "ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137735, "block_size": 4096, "block_total": 240095108, "block_used": 12957373, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986354, "inode_total": 61054976, "inode_used": 68622, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356162560, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}} 2021-02-12T17:54:05.163 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:05 +0000 (0:00:00.068) 0:04:18.555 ******* 2021-02-12T17:54:05.366 INFO:teuthology.task.ansible.out: TASK [testnode : Compile list of non-root partitions] ************************** 2021-02-12T17:54:05.367 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": "lsblk --list --noheadings | grep part | grep -v sda | awk '{ print $1 }'", "delta": "0:00:00.005229", "end": "2021-02-12 17:54:05.339367", "rc": 0, "start": "2021-02-12 17:54:05.334138", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:05.402 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:05 +0000 (0:00:00.238) 0:04:18.793 ******* 2021-02-12T17:54:05.439 INFO:teuthology.task.ansible.out: TASK [testnode : Unmount any non-root mountpoints] ***************************** 2021-02-12T17:54:05.440 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356162560, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137735, 'block_used': 12957373, 'inode_total': 61054976, 'inode_available': 60986354, 'inode_used': 68622, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137735, "block_size": 4096, "block_total": 240095108, "block_used": 12957373, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986354, "inode_total": 61054976, "inode_used": 68622, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356162560, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:05.472 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:05 +0000 (0:00:00.069) 0:04:18.863 ******* 2021-02-12T17:54:05.497 INFO:teuthology.task.ansible.out: TASK [testnode : Wipe filesystems on non-root partitions] ********************** 2021-02-12T17:54:05.528 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:05 +0000 (0:00:00.056) 0:04:18.920 ******* 2021-02-12T17:54:07.367 INFO:teuthology.task.ansible.out: TASK [testnode : Zap all non-root disks] *************************************** 2021-02-12T17:54:07.369 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJdtSMoGf3Q4Cp7GUI2kB1FaTgS21v4NP5'], 'uuids': ['2c14057b-f7f4-443f-bf24-42ca7f308f6c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-1 || sgdisk --zap-all /dev/dm-1", "delta": "0:00:01.625767", "end": "2021-02-12 17:54:07.337630", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJdtSMoGf3Q4Cp7GUI2kB1FaTgS21v4NP5"], "labels": [], "masters": [], "uuids": ["2c14057b-f7f4-443f-bf24-42ca7f308f6c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:05.711863", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-12T17:54:08.645 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-1wKgMl-E0TG-3J25-XpQo-qGhl-moT8-xz1nBE', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400NK400BGN', 'nvme-nvme.8086-504846543632303430304e4b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/nvme0n1 || sgdisk --zap-all /dev/nvme0n1", "delta": "0:00:01.092440", "end": "2021-02-12 17:54:08.618830", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-1wKgMl-E0TG-3J25-XpQo-qGhl-moT8-xz1nBE", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NK400BGN", "nvme-nvme.8086-504846543632303430304e4b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:07.526390", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nGPT data structures destroyed! 2021-02-12T17:54:08.647 INFO:teuthology.task.ansible.out:You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-12T17:54:09.894 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJhLNdBkuT2WNUFfvtjULGxpv6QcH6sKH8'], 'uuids': ['d610608d-efdb-4d90-83ce-ccf92b6215da'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-4 || sgdisk --zap-all /dev/dm-4", "delta": "0:00:01.063363", "end": "2021-02-12 17:54:09.868906", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJhLNdBkuT2WNUFfvtjULGxpv6QcH6sKH8"], "labels": [], "masters": [], "uuids": ["d610608d-efdb-4d90-83ce-ccf92b6215da"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:08.805543", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-12T17:54:11.141 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJe26nftjq3xRp7d3nPuZbc1o0HItFHH8v'], 'uuids': ['d137db07-61d9-4310-8dc0-1b741d9c1e2f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-2 || sgdisk --zap-all /dev/dm-2", "delta": "0:00:01.056100", "end": "2021-02-12 17:54:11.110020", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJe26nftjq3xRp7d3nPuZbc1o0HItFHH8v"], "labels": [], "masters": [], "uuids": ["d137db07-61d9-4310-8dc0-1b741d9c1e2f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:10.053920", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-12T17:54:12.386 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJyVCGGEhGHLyRRI0Go6VieJjhiTH2PJkd'], 'uuids': ['bdb9138c-48ba-4258-8226-516b281475de'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-0 || sgdisk --zap-all /dev/dm-0", "delta": "0:00:01.065554", "end": "2021-02-12 17:54:12.360228", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJyVCGGEhGHLyRRI0Go6VieJjhiTH2PJkd"], "labels": [], "masters": [], "uuids": ["bdb9138c-48ba-4258-8226-516b281475de"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:11.294674", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-12T17:54:12.398 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5R0J2', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5R0J2', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5R0J2', 'scsi-35000c50091e2f7f6', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5R0J2', 'wwn-0x5000c50091e2f7f6'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5R0J2', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e2f7f6', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5R0J2-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5R0J2-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5R0J2-part1', 'scsi-35000c50091e2f7f6-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5R0J2-part1', 'wwn-0x5000c50091e2f7f6-part1'], 'uuids': ['a313f13c-d1e6-43a8-a40a-989733f55ee1'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5R0J2", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5R0J2", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5R0J2", "scsi-35000c50091e2f7f6", "scsi-SATA_ST1000NM0033-9ZM_Z1W5R0J2", "wwn-0x5000c50091e2f7f6"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5R0J2-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5R0J2-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-02-12T17:54:12.400 INFO:teuthology.task.ansible.out:Z1W5R0J2-part1", "scsi-35000c50091e2f7f6-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5R0J2-part1", "wwn-0x5000c50091e2f7f6-part1"], "labels": [], "masters": [], "uuids": ["a313f13c-d1e6-43a8-a40a-989733f55ee1"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5R0J2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2f7f6"}}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:13.650 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJTD22lcFUO7pv61VDZ3Kl9xXj0x1CRuMf'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "sgdisk --zap-all /dev/dm-3 || sgdisk --zap-all /dev/dm-3", "delta": "0:00:01.065377", "end": "2021-02-12 17:54:13.618559", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJTD22lcFUO7pv61VDZ3Kl9xXj0x1CRuMf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:12.553182", "stderr": "", "stderr_lines": [], "stdout": "Creating new GPT entries.\nWarning: The kernel is still using the old partition table.\nThe new table will be used at the next reboot or after you\nrun partprobe(8) or kpartx(8)\nGPT data structures destroyed! You may now partition the disk using fdisk or\nother utilities.", "stdout_lines": ["Creating new GPT entries.", "Warning: The kernel is still using the old partition table.", "The new table will be used at the next reboot or after you", "run partprobe(8) or kpartx(8)", "GPT data structures destroyed! You may now partition the disk using fdisk or", "other utilities."]} 2021-02-12T17:54:13.691 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:13 +0000 (0:00:08.162) 0:04:27.082 ******* 2021-02-12T17:54:14.058 INFO:teuthology.task.ansible.out: TASK [testnode : Blow away lingering OSD data and FSIDs] *********************** 2021-02-12T17:54:14.058 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-1', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_2', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJdtSMoGf3Q4Cp7GUI2kB1FaTgS21v4NP5'], 'uuids': ['2c14057b-f7f4-443f-bf24-42ca7f308f6c'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-1 bs=1M count=110", "delta": "0:00:00.141788", "end": "2021-02-12 17:54:14.032382", "item": {"key": "dm-1", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_2", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJdtSMoGf3Q4Cp7GUI2kB1FaTgS21v4NP5"], "labels": [], "masters": [], "uuids": ["2c14057b-f7f4-443f-bf24-42ca7f308f6c"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:13.890594", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139083 s, 829 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139083 s, 829 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:14.278 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'nvme0n1', 'value': {'virtual': 1, 'links': {'ids': ['lvm-pv-uuid-1wKgMl-E0TG-3J25-XpQo-qGhl-moT8-xz1nBE', 'nvme-INTEL_SSDPEDMD400G4_PHFT620400NK400BGN', 'nvme-nvme.8086-504846543632303430304e4b34303042474e-494e54454c205353445045444d443430304734-00000001'], 'uuids': [], 'labels': [], 'masters': ['dm-0', 'dm-1', 'dm-2', 'dm-3', 'dm-4']}, 'vendor': None, 'model': 'INTEL SSDPEDMD400G4', 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': 'none', 'sectors': '781422768', 'sectorsize': '512', 'size': '372.61 GB', 'host': 'Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)', 'holders': ['vg_nvme-lv_2', 'vg_nvme-lv_5', 'vg_nvme-lv_3', 'vg_nvme-lv_1', 'vg_nvme-lv_4']}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/nvme0n1 bs=1M count=110", "delta": "0:00:00.037635", "end": "2021-02-12 17:54:14.245898", "item": {"key": "nvme0n1", "value": {"holders": ["vg_nvme-lv_2", "vg_nvme-lv_5", "vg_nvme-lv_3", "vg_nvme-lv_1", "vg_nvme-lv_4"], "host": "Non-Volatile memory controller: Intel Corporation PCIe Data Center SSD (rev 01)", "links": {"ids": ["lvm-pv-uuid-1wKgMl-E0TG-3J25-XpQo-qGhl-moT8-xz1nBE", "nvme-INTEL_SSDPEDMD400G4_PHFT620400NK400BGN", "nvme-nvme.8086-504846543632303430304e4b34303042474e-494e54454c205353445045444d443430304734-00000001"], "labels": [], "masters": ["dm-0", "dm-1", "dm-2", "dm-3", "dm-4"], "uuids": []}, "model": "INTEL SSDPEDMD400G4", "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "none", "sectors": "781422768", "sectorsize": "512", "size": "372.61 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:14.208263", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.0350036 s, 3.3 GB/s", "std 2021-02-12T17:54:14.279 INFO:teuthology.task.ansible.out:err_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.0350036 s, 3.3 GB/s"], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:14.601 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-4', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_5', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJhLNdBkuT2WNUFfvtjULGxpv6QcH6sKH8'], 'uuids': ['d610608d-efdb-4d90-83ce-ccf92b6215da'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '31252480', 'sectorsize': '512', 'size': '14.90 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-4 bs=1M count=110", "delta": "0:00:00.143394", "end": "2021-02-12 17:54:14.571376", "item": {"key": "dm-4", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_5", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJhLNdBkuT2WNUFfvtjULGxpv6QcH6sKH8"], "labels": [], "masters": [], "uuids": ["d610608d-efdb-4d90-83ce-ccf92b6215da"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "31252480", "sectorsize": "512", "size": "14.90 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:14.427982", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.140787 s, 819 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.140787 s, 819 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:14.922 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-2', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_3', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJe26nftjq3xRp7d3nPuZbc1o0HItFHH8v'], 'uuids': ['d137db07-61d9-4310-8dc0-1b741d9c1e2f'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-2 bs=1M count=110", "delta": "0:00:00.150909", "end": "2021-02-12 17:54:14.897362", "item": {"key": "dm-2", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_3", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJe26nftjq3xRp7d3nPuZbc1o0HItFHH8v"], "labels": [], "masters": [], "uuids": ["d137db07-61d9-4310-8dc0-1b741d9c1e2f"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:14.746453", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.148376 s, 777 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.148376 s, 777 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:15.245 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-0', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_1', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJyVCGGEhGHLyRRI0Go6VieJjhiTH2PJkd'], 'uuids': ['bdb9138c-48ba-4258-8226-516b281475de'], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-0 bs=1M count=110", "delta": "0:00:00.142063", "end": "2021-02-12 17:54:15.217077", "item": {"key": "dm-0", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_1", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJyVCGGEhGHLyRRI0Go6VieJjhiTH2PJkd"], "labels": [], "masters": [], "uuids": ["bdb9138c-48ba-4258-8226-516b281475de"]}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:15.075014", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.139433 s, 827 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.139433 s, 827 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:15.246 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'key': 'sda', 'value': {'virtual': 1, 'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5R0J2', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5R0J2', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5R0J2', 'scsi-35000c50091e2f7f6', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5R0J2', 'wwn-0x5000c50091e2f7f6'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': 'ATA', 'model': 'ST1000NM0033-9ZM', 'sas_address': None, 'sas_device_handle': None, 'serial': 'Z1W5R0J2', 'removable': '0', 'support_discard': '0', 'wwn': '0x5000c50091e2f7f6', 'partitions': {'sda1': {'links': {'ids': ['ata-ST1000NM0033-9ZM173_Z1W5R0J2-part1', 'scsi-0ATA_ST1000NM0033-9ZM_Z1W5R0J2-part1', 'scsi-1ATA_ST1000NM0033-9ZM173_Z1W5R0J2-part1', 'scsi-35000c50091e2f7f6-part1', 'scsi-SATA_ST1000NM0033-9ZM_Z1W5R0J2-part1', 'wwn-0x5000c50091e2f7f6-part1'], 'uuids': ['a313f13c-d1e6-43a8-a40a-989733f55ee1'], 'labels': [], 'masters': []}, 'start': '2048', 'sectors': '1953522688', 'sectorsize': 512, 'size': '931.51 GB', 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1', 'holders': []}}, 'rotational': '1', 'scheduler_mode': 'mq-deadline', 'sectors': '1953525168', 'sectorsize': '512', 'size': '931.51 GB', 'host': 'SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)', 'holders': []}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "sda", "value": {"holders": [], "host": "SATA controller: Intel Corporation C610/X99 series chipset 6-Port SATA Controller [AHCI mode] (rev 05)", "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5R0J2", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5R0J2", "scsi-1ATA_ST1000NM0033-9ZM173_Z1W5R0J2", "scsi-35000c50091e2f7f6", "scsi-SATA_ST1000NM0033-9ZM_Z1W5R0J2", "wwn-0x5000c50091e2f7f6"], "labels": [], "masters": [], "uuids": []}, "model": "ST1000NM0033-9ZM", "partitions": {"sda1": {"holders": [], "links": {"ids": ["ata-ST1000NM0033-9ZM173_Z1W5R0J2-part1", "scsi-0ATA_ST1000NM0033-9ZM_Z1W5R0J2-part1", "scsi-1ATA_ST1000NM0033-9ZM173_ 2021-02-12T17:54:15.248 INFO:teuthology.task.ansible.out:Z1W5R0J2-part1", "scsi-35000c50091e2f7f6-part1", "scsi-SATA_ST1000NM0033-9ZM_Z1W5R0J2-part1", "wwn-0x5000c50091e2f7f6-part1"], "labels": [], "masters": [], "uuids": ["a313f13c-d1e6-43a8-a40a-989733f55ee1"]}, "sectors": "1953522688", "sectorsize": 512, "size": "931.51 GB", "start": "2048", "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}}, "removable": "0", "rotational": "1", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "mq-deadline", "sectors": "1953525168", "sectorsize": "512", "serial": "Z1W5R0J2", "size": "931.51 GB", "support_discard": "0", "vendor": "ATA", "virtual": 1, "wwn": "0x5000c50091e2f7f6"}}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:15.567 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'dm-3', 'value': {'virtual': 1, 'links': {'ids': ['dm-name-vg_nvme-lv_4', 'dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJTD22lcFUO7pv61VDZ3Kl9xXj0x1CRuMf'], 'uuids': [], 'labels': [], 'masters': []}, 'vendor': None, 'model': None, 'sas_address': None, 'sas_device_handle': None, 'removable': '0', 'support_discard': '512', 'partitions': {}, 'rotational': '0', 'scheduler_mode': '', 'sectors': '187490304', 'sectorsize': '512', 'size': '89.40 GB', 'host': '', 'holders': []}}) => {"ansible_loop_var": "item", "changed": true, "cmd": "dd if=/dev/zero of=/dev/dm-3 bs=1M count=110", "delta": "0:00:00.140723", "end": "2021-02-12 17:54:15.543418", "item": {"key": "dm-3", "value": {"holders": [], "host": "", "links": {"ids": ["dm-name-vg_nvme-lv_4", "dm-uuid-LVM-dG4gNrr0NAneZvfwSAxw8NH2nVywAVlJTD22lcFUO7pv61VDZ3Kl9xXj0x1CRuMf"], "labels": [], "masters": [], "uuids": []}, "model": null, "partitions": {}, "removable": "0", "rotational": "0", "sas_address": null, "sas_device_handle": null, "scheduler_mode": "", "sectors": "187490304", "sectorsize": "512", "size": "89.40 GB", "support_discard": "512", "vendor": null, "virtual": 1}}, "rc": 0, "start": "2021-02-12 17:54:15.402695", "stderr": "110+0 records in\n110+0 records out\n115343360 bytes (115 MB, 110 MiB) copied, 0.138103 s, 835 MB/s", "stderr_lines": ["110+0 records in", "110+0 records out", "115343360 bytes (115 MB, 110 MiB) copied, 0.138103 s, 835 MB/s"], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:15.606 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:15 +0000 (0:00:01.914) 0:04:28.997 ******* 2021-02-12T17:54:15.870 INFO:teuthology.task.ansible.out: TASK [testnode : Remove all LVM data] ****************************************** 2021-02-12T17:54:15.872 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"attempts": 1, "changed": true, "cmd": "dmsetup remove_all --force", "delta": "0:00:00.072710", "end": "2021-02-12 17:54:15.841534", "rc": 0, "start": "2021-02-12 17:54:15.768824", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:15.904 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:15 +0000 (0:00:00.297) 0:04:29.295 ******* 2021-02-12T17:54:16.143 INFO:teuthology.task.ansible.out: TASK [testnode : Check for physical volumes] *********************************** 2021-02-12T17:54:16.144 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": "pvdisplay | grep 'PV Name' | awk '{ print $3 }'", "delta": "0:00:00.056760", "end": "2021-02-12 17:54:16.124224", "rc": 0, "start": "2021-02-12 17:54:16.067464", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:16.179 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.274) 0:04:29.570 ******* 2021-02-12T17:54:16.205 INFO:teuthology.task.ansible.out: TASK [testnode : Remove physical volumes] ************************************** 2021-02-12T17:54:16.239 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.060) 0:04:29.630 ******* 2021-02-12T17:54:16.265 INFO:teuthology.task.ansible.out: TASK [testnode : Write a new partition table to {{ item.value.device }}] ******* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:16.301 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.061) 0:04:29.692 ******* 2021-02-12T17:54:16.327 INFO:teuthology.task.ansible.out: TASK [testnode : Write partition entries to {{ item.0.device }}] *************** 2021-02-12T17:54:16.328 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:16.361 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.059) 0:04:29.752 ******* 2021-02-12T17:54:16.387 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-12T17:54:16.389 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:16.423 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.062) 0:04:29.814 ******* 2021-02-12T17:54:16.449 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs for {{ item.0.device }}] ****************** 2021-02-12T17:54:16.451 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:16.486 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.062) 0:04:29.877 ******* 2021-02-12T17:54:16.668 INFO:teuthology.task.ansible.out: TASK [testnode : Edit lvm.conf to support LVM on kRBD.] ************************ 2021-02-12T17:54:16.670 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": false, "msg": ""} 2021-02-12T17:54:16.703 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.216) 0:04:30.094 ******* 2021-02-12T17:54:16.739 INFO:teuthology.task.ansible.out: TASK [testnode : Set root disk] ************************************************ 2021-02-12T17:54:16.740 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'mount': '/', 'device': '/dev/sda1', 'fstype': 'ext4', 'options': 'rw,seclabel,relatime', 'size_total': 983429562368, 'size_available': 930356162560, 'block_size': 4096, 'block_total': 240095108, 'block_available': 227137735, 'block_used': 12957373, 'inode_total': 61054976, 'inode_available': 60986354, 'inode_used': 68622, 'uuid': 'a313f13c-d1e6-43a8-a40a-989733f55ee1'}) => {"ansible_loop_var": "item", "changed": false, "item": {"block_available": 227137735, "block_size": 4096, "block_total": 240095108, "block_used": 12957373, "device": "/dev/sda1", "fstype": "ext4", "inode_available": 60986354, "inode_total": 61054976, "inode_used": 68622, "mount": "/", "options": "rw,seclabel,relatime", "size_available": 930356162560, "size_total": 983429562368, "uuid": "a313f13c-d1e6-43a8-a40a-989733f55ee1"}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:16.774 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.071) 0:04:30.165 ******* 2021-02-12T17:54:16.800 INFO:teuthology.task.ansible.out: TASK [testnode : Combine list of non-root disks] ******************************* 2021-02-12T17:54:16.802 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:16.834 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.059) 0:04:30.225 ******* 2021-02-12T17:54:16.860 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-12T17:54:16.861 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:16.893 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.059) 0:04:30.284 ******* 2021-02-12T17:54:16.919 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:16.953 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:16 +0000 (0:00:00.059) 0:04:30.344 ******* 2021-02-12T17:54:16.978 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume_groups dict] ************************************ 2021-02-12T17:54:16.980 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:17.013 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:17 +0000 (0:00:00.059) 0:04:30.404 ******* 2021-02-12T17:54:17.039 INFO:teuthology.task.ansible.out: TASK [testnode : Determine desired logical volume percentage size] ************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:17.072 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:17 +0000 (0:00:00.059) 0:04:30.463 ******* 2021-02-12T17:54:17.098 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical_volumes dict] ********************************** 2021-02-12T17:54:17.099 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:17.132 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:17 +0000 (0:00:00.059) 0:04:30.523 ******* 2021-02-12T17:54:17.843 INFO:teuthology.task.ansible.out: TASK [testnode : Create volume group(s)] *************************************** 2021-02-12T17:54:17.844 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'vg_nvme', 'value': {'pvs': '/dev/nvme0n1'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "vg_nvme", "value": {"pvs": "/dev/nvme0n1"}}} 2021-02-12T17:54:17.877 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:17 +0000 (0:00:00.745) 0:04:31.268 ******* 2021-02-12T17:54:18.525 INFO:teuthology.task.ansible.out: TASK [testnode : Create logical volume(s)] ************************************* 2021-02-12T17:54:18.526 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-12T17:54:18.868 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-12T17:54:19.189 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-12T17:54:19.552 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-12T17:54:19.883 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": true, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "msg": ""} 2021-02-12T17:54:19.917 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:19 +0000 (0:00:02.040) 0:04:33.309 ******* 2021-02-12T17:54:20.109 INFO:teuthology.task.ansible.out: TASK [testnode : Erase /scratch_devs so we know it's accurate] ***************** 2021-02-12T17:54:20.110 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "path": "/scratch_devs", "state": "absent"} 2021-02-12T17:54:20.143 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:20 +0000 (0:00:00.225) 0:04:33.534 ******* 2021-02-12T17:54:20.349 INFO:teuthology.task.ansible.out: TASK [testnode : Write /scratch_devs] ****************************************** 2021-02-12T17:54:20.350 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_1', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_1", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-12T17:54:20.530 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_2', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_2", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-12T17:54:20.696 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_3', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_3", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-12T17:54:20.875 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_4', 'value': {'vg': 'vg_nvme', 'size': '89.4G', 'scratch_dev': True}}) => {"ansible_loop_var": "item", "backup": "", "changed": true, "item": {"key": "lv_4", "value": {"scratch_dev": true, "size": "89.4G", "vg": "vg_nvme"}}, "msg": "line added"} 2021-02-12T17:54:20.886 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'key': 'lv_5', 'value': {'vg': 'vg_nvme', 'size': '14.9G'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lv_5", "value": {"size": "14.9G", "vg": "vg_nvme"}}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:20.923 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:20 +0000 (0:00:00.779) 0:04:34.314 ******* 2021-02-12T17:54:20.951 INFO:teuthology.task.ansible.out: TASK [testnode : Create filesystems] ******************************************* 2021-02-12T17:54:20.952 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:20.985 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:20 +0000 (0:00:00.061) 0:04:34.376 ******* 2021-02-12T17:54:21.011 INFO:teuthology.task.ansible.out: TASK [testnode : Mount filesystems] ******************************************** 2021-02-12T17:54:21.012 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:21.044 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:21 +0000 (0:00:00.059) 0:04:34.436 ******* 2021-02-12T17:54:21.236 INFO:teuthology.task.ansible.out: TASK [testnode : Create /var/lib/ceph] ***************************************** 2021-02-12T17:54:21.238 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 0, "group": "root", "mode": "0755", "owner": "root", "path": "/var/lib/ceph", "secontext": "unconfined_u:object_r:var_lib_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-12T17:54:21.270 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:21 +0000 (0:00:00.225) 0:04:34.662 ******* 2021-02-12T17:54:21.309 INFO:teuthology.task.ansible.out: TASK [testnode : Set xfs_opts on newer OSes] *********************************** 2021-02-12T17:54:21.311 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"xfs_opts": "-m crc=0,finobt=0"}, "changed": false} 2021-02-12T17:54:21.344 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:21 +0000 (0:00:00.073) 0:04:34.735 ******* 2021-02-12T17:54:21.857 INFO:teuthology.task.ansible.out: TASK [testnode : Create xfs filesystem on /dev/vg_nvme/lv_5] ******************* 2021-02-12T17:54:21.858 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true} 2021-02-12T17:54:21.892 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:21 +0000 (0:00:00.548) 0:04:35.284 ******* 2021-02-12T17:54:22.447 INFO:teuthology.task.ansible.out: TASK [testnode : Mount /dev/vg_nvme/lv_5 to /var/lib/ceph] ********************* 2021-02-12T17:54:22.448 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "dump": "0", "fstab": "/etc/fstab", "fstype": "xfs", "name": "/var/lib/ceph", "opts": "defaults,nofail", "passno": "0", "src": "/dev/vg_nvme/lv_5"} 2021-02-12T17:54:22.481 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:22 +0000 (0:00:00.588) 0:04:35.872 ******* 2021-02-12T17:54:22.508 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd] ****************************************** 2021-02-12T17:54:22.509 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:22.544 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:22 +0000 (0:00:00.062) 0:04:35.935 ******* 2021-02-12T17:54:22.811 INFO:teuthology.task.ansible.out: TASK [testnode : Install cachefilesd conf file] ******************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 12 February 2021 17:54:22 +0000 (0:00:00.059) 0:04:35.994 ******* TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 12 February 2021 17:54:22 +0000 (0:00:00.058) 0:04:36.053 ******* TASK [testnode : Restart cachefilesd] ****************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 12 February 2021 17:54:22 +0000 (0:00:00.061) 0:04:36.115 ******* TASK [testnode : Add CPAN config directory for the teuthology user.] *********** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} Friday 12 February 2021 17:54:22 +0000 (0:00:00.059) 0:04:36.174 ******* TASK [testnode : Add CPAN config directory for the root user.] ***************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:22.843 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:22 +0000 (0:00:00.059) 0:04:36.234 ******* 2021-02-12T17:54:22.869 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for the teuthology user.] ****************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:22.902 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:22 +0000 (0:00:00.059) 0:04:36.293 ******* 2021-02-12T17:54:22.930 INFO:teuthology.task.ansible.out: TASK [testnode : Upload CPAN config for root.] ********************************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:22.962 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:22 +0000 (0:00:00.060) 0:04:36.354 ******* 2021-02-12T17:54:22.988 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure perl-doc is installed on apt systems.] ***************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:23.022 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:23 +0000 (0:00:00.059) 0:04:36.413 ******* 2021-02-12T17:54:23.048 INFO:teuthology.task.ansible.out: TASK [testnode : Check to see if Amazon::S3 is installed.] ********************* 2021-02-12T17:54:23.049 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:23.082 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:23 +0000 (0:00:00.060) 0:04:36.473 ******* 2021-02-12T17:54:23.119 INFO:teuthology.task.ansible.out: TASK [testnode : Install Amazon::S3.] ****************************************** 2021-02-12T17:54:23.120 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:23.153 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:23 +0000 (0:00:00.070) 0:04:36.544 ******* 2021-02-12T17:54:23.178 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on rpm based systems.] ******************** 2021-02-12T17:54:23.180 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:23.211 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:23 +0000 (0:00:00.058) 0:04:36.602 ******* 2021-02-12T17:54:23.237 INFO:teuthology.task.ansible.out: TASK [testnode : Install ntp package on apt based systems.] ******************** 2021-02-12T17:54:23.238 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:23.269 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:23 +0000 (0:00:00.058) 0:04:36.661 ******* 2021-02-12T17:54:23.296 INFO:teuthology.task.ansible.out: TASK [testnode : Remove racy /etc/network/if-up.d/ntpdate on xenial] *********** 2021-02-12T17:54:23.298 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:23.328 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:23 +0000 (0:00:00.058) 0:04:36.720 ******* 2021-02-12T17:54:23.354 INFO:teuthology.task.ansible.out: TASK [testnode : Create the ntp.conf file.] ************************************ skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:23.388 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:23 +0000 (0:00:00.059) 0:04:36.779 ******* 2021-02-12T17:54:23.847 INFO:teuthology.task.ansible.out: TASK [testnode : Create the chrony.conf file] ********************************** 2021-02-12T17:54:23.849 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "a7ecef00073feab34f868065d34e9dac55134241", "dest": "/etc/chrony.conf", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "path": "/etc/chrony.conf", "secontext": "system_u:object_r:etc_t:s0", "size": 215, "state": "file", "uid": 0} 2021-02-12T17:54:23.882 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:23 +0000 (0:00:00.493) 0:04:37.273 ******* 2021-02-12T17:54:24.178 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ntpd is running.] *********************************** 2021-02-12T17:54:24.179 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "enabled": true, "name": "chronyd", "state": "started", "status": {"ActiveEnterTimestamp": "Fri 2021-02-12 17:49:06 UTC", "ActiveEnterTimestampMonotonic": "13021441", "ActiveExitTimestampMonotonic": "0", "ActiveState": "active", "After": "ntpd.service systemd-journald.socket basic.target system.slice tmp.mount ntpdate.service -.mount systemd-tmpfiles-setup.service sysinit.target sntp.service", "AllowIsolate": "no", "AllowedCPUs": "", "AllowedMemoryNodes": "", "AmbientCapabilities": "", "AssertResult": "yes", "AssertTimestamp": "Fri 2021-02-12 17:49:05 UTC", "AssertTimestampMonotonic": "11689542", "Before": "shutdown.target multi-user.target", "BlockIOAccounting": "no", "BlockIOWeight": "[not set]", "CPUAccounting": "no", "CPUAffinity": "", "CPUQuotaPerSecUSec": "infinity", "CPUSchedulingPolicy": "0", "CPUSchedulingPriority": "0", "CPUSchedulingResetOnFork": "no", "CPUShares": "[not set]", "CPUUsageNSec": "[not set]", "CPUWeight": "[not set]", "CacheDirectoryMode": "0755", "CanIsolate": "no", "CanReload": "no", "CanStart": "yes", "CanStop": "yes", "CapabilityBoundingSet": "cap_chown cap_dac_override cap_dac_read_search cap_fowner cap_fsetid cap_kill cap_setgid cap_setuid cap_setpcap cap_linux_immutable cap_net_bind_service cap_net_broadcast cap_net_admin cap_net_raw cap_ipc_lock cap_ipc_owner cap_sys_module cap_sys_rawio cap_sys_chroot cap_sys_ptrace cap_sys_pacct cap_sys_admin cap_sys_boot cap_sys_nice cap_sys_resource cap_sys_time cap_sys_tty_config cap_mknod cap_lease cap_audit_write cap_audit_control cap_setfcap cap_mac_override cap_mac_admin cap_syslog cap_wake_alarm cap_block_suspend", "CollectMode": "inactive", "ConditionResult": "yes", "ConditionTimestamp": "Fri 2021-02-12 17:49:05 UTC", "ConditionTimestampMonotonic": "11689500", "ConfigurationDirectoryMode": "0755", "Conflicts": "shutdown.target systemd-timesyncd.service ntpd.service", "ControlGroup": "/system.slice/chronyd.service", "ControlPID": "0 2021-02-12T17:54:24.180 INFO:teuthology.task.ansible.out:", "DefaultDependencies": "yes", "Delegate": "no", "Description": "NTP client/server", "DevicePolicy": "auto", "Documentation": "man:chronyd(8) man:chrony.conf(5)", "DynamicUser": "no", "EffectiveCPUs": "", "EffectiveMemoryNodes": "", "EnvironmentFiles": "/etc/sysconfig/chronyd (ignore_errors=yes)", "ExecMainCode": "0", "ExecMainExitTimestampMonotonic": "0", "ExecMainPID": "883", "ExecMainStartTimestamp": "Fri 2021-02-12 17:49:06 UTC", "ExecMainStartTimestampMonotonic": "12960024", "ExecMainStatus": "0", "ExecStart": "{ path=/usr/sbin/chronyd ; argv[]=/usr/sbin/chronyd $OPTIONS ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "ExecStartPost": "{ path=/usr/libexec/chrony-helper ; argv[]=/usr/libexec/chrony-helper update-daemon ; ignore_errors=no ; start_time=[n/a] ; stop_time=[n/a] ; pid=0 ; code=(null) ; status=0/0 }", "FailureAction": "none", "FileDescriptorStoreMax": "0", "FragmentPath": "/usr/lib/systemd/system/chronyd.service", "GID": "[not set]", "GuessMainPID": "yes", "IOAccounting": "no", "IOSchedulingClass": "0", "IOSchedulingPriority": "0", "IOWeight": "[not set]", "IPAccounting": "no", "IPEgressBytes": "18446744073709551615", "IPEgressPackets": "18446744073709551615", "IPIngressBytes": "18446744073709551615", "IPIngressPackets": "18446744073709551615", "Id": "chronyd.service", "IgnoreOnIsolate": "no", "IgnoreSIGPIPE": "yes", "InactiveEnterTimestampMonotonic": "0", "InactiveExitTimestamp": "Fri 2021-02-12 17:49:05 UTC", "InactiveExitTimestampMonotonic": "11704110", "InvocationID": "b150098d737e4852a2ea4d1aa4548a73", "JobRunningTimeoutUSec": "infinity", "JobTimeoutAction": "none", "JobTimeoutUSec": "infinity", "KeyringMode": "private", "KillMode": "control-group", "KillSignal": "15", "LimitAS": "infinity", "LimitASSoft": "infinity", "LimitCORE": "infinity", "LimitCORESoft": "infinity", "LimitCPU": "infinity", "LimitCPUSoft": "infinity", "LimitDATA": "infinity", "LimitDATASoft": "infinity", "LimitFSIZE": "infinity 2021-02-12T17:54:24.181 INFO:teuthology.task.ansible.out:", "LimitFSIZESoft": "infinity", "LimitLOCKS": "infinity", "LimitLOCKSSoft": "infinity", "LimitMEMLOCK": "65536", "LimitMEMLOCKSoft": "65536", "LimitMSGQUEUE": "819200", "LimitMSGQUEUESoft": "819200", "LimitNICE": "0", "LimitNICESoft": "0", "LimitNOFILE": "262144", "LimitNOFILESoft": "1024", "LimitNPROC": "127260", "LimitNPROCSoft": "127260", "LimitRSS": "infinity", "LimitRSSSoft": "infinity", "LimitRTPRIO": "0", "LimitRTPRIOSoft": "0", "LimitRTTIME": "infinity", "LimitRTTIMESoft": "infinity", "LimitSIGPENDING": "127260", "LimitSIGPENDINGSoft": "127260", "LimitSTACK": "infinity", "LimitSTACKSoft": "8388608", "LoadState": "loaded", "LockPersonality": "no", "LogLevelMax": "-1", "LogRateLimitBurst": "0", "LogRateLimitIntervalUSec": "0", "LogsDirectoryMode": "0755", "MainPID": "883", "MemoryAccounting": "yes", "MemoryCurrent": "3559424", "MemoryDenyWriteExecute": "no", "MemoryHigh": "infinity", "MemoryLimit": "infinity", "MemoryLow": "0", "MemoryMax": "infinity", "MemorySwapMax": "infinity", "MountAPIVFS": "no", "MountFlags": "", "NFileDescriptorStore": "0", "NRestarts": "0", "NUMAMask": "", "NUMAPolicy": "n/a", "Names": "chronyd.service", "NeedDaemonReload": "no", "Nice": "0", "NoNewPrivileges": "no", "NonBlocking": "no", "NotifyAccess": "none", "OOMScoreAdjust": "0", "OnFailureJobMode": "replace", "PIDFile": "/run/chrony/chronyd.pid", "PermissionsStartOnly": "no", "Perpetual": "no", "PrivateDevices": "no", "PrivateMounts": "no", "PrivateNetwork": "no", "PrivateTmp": "yes", "PrivateUsers": "no", "ProtectControlGroups": "no", "ProtectHome": "yes", "ProtectKernelModules": "no", "ProtectKernelTunables": "no", "ProtectSystem": "full", "RefuseManualStart": "no", "RefuseManualStop": "no", "RemainAfterExit": "no", "RemoveIPC": "no", "Requires": "-.mount system.slice sysinit.target", "RequiresMountsFor": "/var/tmp", "Restart": "no", "RestartUSec": "100ms", "RestrictNamespaces": "no", "RestrictRealtime": "no", "RestrictSUIDSGID": "no", "Result": "success", "RootDirectoryStartOn 2021-02-12T17:54:24.183 INFO:teuthology.task.ansible.out:ly": "no", "RuntimeDirectoryMode": "0755", "RuntimeDirectoryPreserve": "no", "RuntimeMaxUSec": "infinity", "SameProcessGroup": "no", "SecureBits": "0", "SendSIGHUP": "no", "SendSIGKILL": "yes", "Slice": "system.slice", "StandardError": "inherit", "StandardInput": "null", "StandardInputData": "", "StandardOutput": "journal", "StartLimitAction": "none", "StartLimitBurst": "5", "StartLimitIntervalUSec": "10s", "StartupBlockIOWeight": "[not set]", "StartupCPUShares": "[not set]", "StartupCPUWeight": "[not set]", "StartupIOWeight": "[not set]", "StateChangeTimestamp": "Fri 2021-02-12 17:49:06 UTC", "StateChangeTimestampMonotonic": "13021441", "StateDirectoryMode": "0755", "StatusErrno": "0", "StopWhenUnneeded": "no", "SubState": "running", "SuccessAction": "none", "SyslogFacility": "3", "SyslogLevel": "6", "SyslogLevelPrefix": "yes", "SyslogPriority": "30", "SystemCallErrorNumber": "0", "TTYReset": "no", "TTYVHangup": "no", "TTYVTDisallocate": "no", "TasksAccounting": "yes", "TasksCurrent": "1", "TasksMax": "203616", "TimeoutStartUSec": "1min 30s", "TimeoutStopUSec": "1min 30s", "TimerSlackNSec": "50000", "Transient": "no", "Type": "forking", "UID": "[not set]", "UMask": "0022", "UnitFilePreset": "enabled", "UnitFileState": "enabled", "UtmpMode": "init", "WantedBy": "multi-user.target", "WatchdogTimestamp": "Fri 2021-02-12 17:49:06 UTC", "WatchdogTimestampMonotonic": "12960036", "WatchdogUSec": "0"}} 2021-02-12T17:54:24.213 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:24 +0000 (0:00:00.331) 0:04:37.605 ******* 2021-02-12T17:54:24.239 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-12T17:54:24.240 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"pip_executable": "pip", "pip_version": "python-pip"}, "changed": false} 2021-02-12T17:54:24.273 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:24 +0000 (0:00:00.059) 0:04:37.664 ******* 2021-02-12T17:54:24.310 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-12T17:54:24.311 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"ansible_python_interpreter": "/usr/bin/python3", "pip_executable": "pip3", "pip_version": "python3-pip"}, "changed": false} 2021-02-12T17:54:24.347 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:24 +0000 (0:00:00.073) 0:04:37.738 ******* 2021-02-12T17:54:24.373 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on rpm based systems.] ********************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:24.406 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:24 +0000 (0:00:00.059) 0:04:37.797 ******* 2021-02-12T17:54:24.432 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on apt based systems.] ********************* skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:24.464 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:24 +0000 (0:00:00.057) 0:04:37.855 ******* 2021-02-12T17:54:24.488 INFO:teuthology.task.ansible.out: TASK [testnode : Install python-pip on zypper based systems.] ****************** 2021-02-12T17:54:24.490 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:24.520 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:24 +0000 (0:00:00.056) 0:04:37.912 ******* 2021-02-12T17:54:24.711 INFO:teuthology.task.ansible.out: TASK [testnode : Create the .pip directory for the teuthology user.] *********** 2021-02-12T17:54:24.713 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 4096, "state": "directory", "uid": 1000} 2021-02-12T17:54:24.745 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:24 +0000 (0:00:00.224) 0:04:38.137 ******* 2021-02-12T17:54:25.215 INFO:teuthology.task.ansible.out: TASK [testnode : Create pip.conf and configure it to use our mirror] *********** 2021-02-12T17:54:25.217 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "checksum": "959d37ab30545da205b389a6b7561b225a478a06", "dest": "/home/ubuntu/.pip/pip.conf", "gid": 1236, "group": "ubuntu", "mode": "0644", "owner": "ubuntu", "path": "/home/ubuntu/.pip/pip.conf", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 73, "state": "file", "uid": 1000} 2021-02-12T17:54:25.249 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:25 +0000 (0:00:00.503) 0:04:38.640 ******* 2021-02-12T17:54:26.570 INFO:teuthology.task.ansible.out: TASK [testnode : Install packages via pip] ************************************* 2021-02-12T17:54:26.572 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "cmd": ["/bin/pip3", "install", "remoto>=0.0.35"], "name": ["remoto>=0.0.35"], "requirements": null, "state": "present", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages\nRequirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)\nRequirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)\n", "stdout_lines": ["Requirement already satisfied: remoto>=0.0.35 in /usr/local/lib/python3.6/site-packages", "Requirement already satisfied: execnet in /usr/local/lib/python3.6/site-packages (from remoto>=0.0.35)", "Requirement already satisfied: apipkg>=1.4 in /usr/local/lib/python3.6/site-packages (from execnet->remoto>=0.0.35)"], "version": null, "virtualenv": null} 2021-02-12T17:54:26.604 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:26 +0000 (0:00:01.355) 0:04:39.996 ******* 2021-02-12T17:54:26.631 INFO:teuthology.task.ansible.out: TASK [testnode : Purge resolvconf] ********************************************* 2021-02-12T17:54:26.632 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:26.664 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:26 +0000 (0:00:00.059) 0:04:40.055 ******* 2021-02-12T17:54:26.690 INFO:teuthology.task.ansible.out: TASK [testnode : Set interface] ************************************************ 2021-02-12T17:54:26.691 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"interface": "enp3s0f1"}, "changed": false} 2021-02-12T17:54:26.726 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:26 +0000 (0:00:00.061) 0:04:40.117 ******* 2021-02-12T17:54:26.906 INFO:teuthology.task.ansible.out: TASK [testnode : Check for presence of /etc/network/interfaces] **************** 2021-02-12T17:54:26.907 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "stat": {"exists": false}} 2021-02-12T17:54:26.939 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:26 +0000 (0:00:00.213) 0:04:40.330 ******* 2021-02-12T17:54:26.965 INFO:teuthology.task.ansible.out: TASK [testnode : Rewrite /etc/network/interfaces to use dhcp] ****************** 2021-02-12T17:54:26.967 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.000 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:26 +0000 (0:00:00.061) 0:04:40.391 ******* 2021-02-12T17:54:27.026 INFO:teuthology.task.ansible.out: TASK [testnode : Set bounce_interface if we just enabled dhcp] ***************** 2021-02-12T17:54:27.028 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"bounce_interface": false}, "changed": false} 2021-02-12T17:54:27.060 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.060) 0:04:40.452 ******* 2021-02-12T17:54:27.088 INFO:teuthology.task.ansible.out: TASK [testnode : ifdown and ifup] ********************************************** 2021-02-12T17:54:27.089 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.121 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.060) 0:04:40.512 ******* 2021-02-12T17:54:27.304 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure lab_domain is in search domains in /etc/resolv.conf] *** 2021-02-12T17:54:27.306 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line replaced"} 2021-02-12T17:54:27.339 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.218) 0:04:40.730 ******* 2021-02-12T17:54:27.521 INFO:teuthology.task.ansible.out: TASK [testnode : Ensure domain is set in /etc/resolv.conf] ********************* 2021-02-12T17:54:27.522 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"backup": "", "changed": true, "msg": "line added"} 2021-02-12T17:54:27.553 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.214) 0:04:40.945 ******* 2021-02-12T17:54:27.579 INFO:teuthology.task.ansible.out: TASK [testnode : Make sure ethtool is installed] ******************************* 2021-02-12T17:54:27.580 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.612 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.058) 0:04:41.003 ******* 2021-02-12T17:54:27.648 INFO:teuthology.task.ansible.out: TASK [testnode : grep ethtool for secondary NIC MAC address] ******************* 2021-02-12T17:54:27.649 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=enp3s0f0) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=eno1) => {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item=eno2) => {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.660 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=enp3s0f1) => {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.661 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item=lo) => {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.694 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.081) 0:04:41.085 ******* 2021-02-12T17:54:27.730 INFO:teuthology.task.ansible.out: TASK [testnode : Define net_to_configure var] ********************************** 2021-02-12T17:54:27.732 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f0', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f0", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.742 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'eno2', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "eno2", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.743 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'enp3s0f1', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "enp3s0f1", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.744 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => (item={'changed': False, 'skipped': True, 'skip_reason': 'Conditional result was False', 'item': 'lo', 'ansible_loop_var': 'item'}) => {"ansible_loop_var": "item", "changed": false, "item": {"ansible_loop_var": "item", "changed": false, "item": "lo", "skip_reason": "Conditional result was False", "skipped": true}, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.776 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.082) 0:04:41.168 ******* 2021-02-12T17:54:27.804 INFO:teuthology.task.ansible.out: TASK [testnode : set_fact] ***************************************************** 2021-02-12T17:54:27.806 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.837 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.060) 0:04:41.228 ******* 2021-02-12T17:54:27.863 INFO:teuthology.task.ansible.out: TASK [testnode : Check if {{ nic_to_configure }} is 10Gb] ********************** 2021-02-12T17:54:27.864 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.895 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.058) 0:04:41.286 ******* 2021-02-12T17:54:27.921 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU to 9000 if 10Gb] ************************************** 2021-02-12T17:54:27.923 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:27.956 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:27 +0000 (0:00:00.060) 0:04:41.347 ******* 2021-02-12T17:54:27.986 INFO:teuthology.task.ansible.out: TASK [testnode : Write Ubuntu network config for {{ nic_to_configure }}] ******* 2021-02-12T17:54:27.987 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:28.019 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:28 +0000 (0:00:00.062) 0:04:41.410 ******* 2021-02-12T17:54:28.184 INFO:teuthology.task.ansible.out: TASK [testnode : Set MTU on Ubuntu] ******************************************** 2021-02-12T17:54:28.185 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:28.218 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:28 +0000 (0:00:00.199) 0:04:41.609 ******* 2021-02-12T17:54:28.244 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-12T17:54:28.245 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:28.282 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:28 +0000 (0:00:00.064) 0:04:41.673 ******* 2021-02-12T17:54:28.309 INFO:teuthology.task.ansible.out: TASK [testnode : Write RHEL/CentOS network config for {{ nic_to_configure }}] *** 2021-02-12T17:54:28.311 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:28.346 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:28 +0000 (0:00:00.063) 0:04:41.737 ******* 2021-02-12T17:54:28.373 INFO:teuthology.task.ansible.out: TASK [testnode : Bounce {{ nic_to_configure }}] ******************************** 2021-02-12T17:54:28.374 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:28.407 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:28 +0000 (0:00:00.060) 0:04:41.798 ******* 2021-02-12T17:54:28.619 INFO:teuthology.task.ansible.out: TASK [testnode : List any leftover Ceph artifacts from previous jobs] ********** 2021-02-12T17:54:28.621 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/var/run/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/run/ -name \"*ceph*\"", "delta": "0:00:00.005018", "end": "2021-02-12 17:54:28.591613", "failed_when_result": false, "item": "/var/run/", "rc": 0, "start": "2021-02-12 17:54:28.586595", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:28.796 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/etc/systemd/system/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/systemd/system/ -name \"*ceph*\"", "delta": "0:00:00.003054", "end": "2021-02-12 17:54:28.768604", "failed_when_result": false, "item": "/etc/systemd/system/", "rc": 0, "start": "2021-02-12 17:54:28.765550", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:28.972 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/etc/ceph) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /etc/ceph -name \"*ceph*\"", "delta": "0:00:00.002755", "end": "2021-02-12 17:54:28.940830", "failed_when_result": false, "item": "/etc/ceph", "msg": "non-zero return code", "rc": 1, "start": "2021-02-12 17:54:28.938075", "stderr": "find: ‘/etc/ceph’: No such file or directory", "stderr_lines": ["find: ‘/etc/ceph’: No such file or directory"], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:29.137 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/var/log/) => {"ansible_loop_var": "item", "changed": false, "cmd": "find /var/log/ -name \"*ceph*\"", "delta": "0:00:00.003163", "end": "2021-02-12 17:54:29.113870", "failed_when_result": false, "item": "/var/log/", "rc": 0, "start": "2021-02-12 17:54:29.110707", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:29.172 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:29 +0000 (0:00:00.765) 0:04:42.563 ******* 2021-02-12T17:54:29.197 INFO:teuthology.task.ansible.out: TASK [testnode : Delete any leftover Ceph artifacts from previous jobs] ******** 2021-02-12T17:54:29.229 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:29 +0000 (0:00:00.057) 0:04:42.621 ******* 2021-02-12T17:54:29.411 INFO:teuthology.task.ansible.out: TASK [testnode : Touch /ceph-qa-ready] ***************************************** 2021-02-12T17:54:29.412 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "dest": "/ceph-qa-ready", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:etc_runtime_t:s0", "size": 0, "state": "file", "uid": 0} 2021-02-12T17:54:29.513 INFO:teuthology.task.ansible.out: PLAY [testnodes,senta,vossi,jenkins_slaves] ************************************ 2021-02-12T17:54:29.548 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:29 +0000 (0:00:00.318) 0:04:42.939 ******* 2021-02-12T17:54:30.185 INFO:teuthology.task.ansible.out: TASK [Gathering Facts] ********************************************************* ok: [smithi107.front.sepia.ceph.com] 2021-02-12T17:54:30.258 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:30 +0000 (0:00:00.710) 0:04:43.649 ******* 2021-02-12T17:54:30.284 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:30.318 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:30 +0000 (0:00:00.060) 0:04:43.709 ******* 2021-02-12T17:54:30.344 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-12T17:54:30.346 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"package_manager": "yum"}, "changed": false} 2021-02-12T17:54:30.379 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:30 +0000 (0:00:00.060) 0:04:43.770 ******* 2021-02-12T17:54:30.415 INFO:teuthology.task.ansible.out: TASK [container-host : Including distro specific variables] ******************** 2021-02-12T17:54:30.417 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => (item=/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml) => {"ansible_facts": {"container_packages": ["podman", "podman-docker"]}, "ansible_included_var_files": ["/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"], "ansible_loop_var": "item", "changed": false, "item": "/home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_master/roles/container-host/vars/yum_systems.yml"} 2021-02-12T17:54:30.450 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:30 +0000 (0:00:00.070) 0:04:43.841 ******* 2021-02-12T17:54:31.500 INFO:teuthology.task.ansible.out: TASK [container-host : Install container packages] ***************************** 2021-02-12T17:54:31.502 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"changed": false, "msg": "Nothing to do", "rc": 0, "results": ["Installed: podman", "Installed: podman-docker"]} 2021-02-12T17:54:31.538 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:31 +0000 (0:00:01.088) 0:04:44.929 ******* 2021-02-12T17:54:31.564 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-12T17:54:31.565 INFO:teuthology.task.ansible.out:ok: [smithi107.front.sepia.ceph.com] => {"ansible_facts": {"container_service_conf": "/etc/containers/registries.conf"}, "changed": false} 2021-02-12T17:54:31.600 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:31 +0000 (0:00:00.062) 0:04:44.991 ******* 2021-02-12T17:54:31.627 INFO:teuthology.task.ansible.out: TASK [container-host : set_fact] *********************************************** 2021-02-12T17:54:31.628 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:31.663 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:31 +0000 (0:00:00.062) 0:04:45.054 ******* 2021-02-12T17:54:31.875 INFO:teuthology.task.ansible.out: TASK [container-host : Create container_mirror_cert_paths] ********************* 2021-02-12T17:54:31.876 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-12T17:54:32.062 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "mode": "0755", "owner": "root", "path": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "secontext": "unconfined_u:object_r:etc_t:s0", "size": 4096, "state": "directory", "uid": 0} 2021-02-12T17:54:32.099 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:32 +0000 (0:00:00.436) 0:04:45.490 ******* 2021-02-12T17:54:32.598 INFO:teuthology.task.ansible.out: TASK [container-host : Copy docker-mirror.front.sepia.ceph.com:5000 self-signed cert] *** 2021-02-12T17:54:32.600 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item=/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/docker/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:cert_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613152472.1345146-167740296071855/source", "state": "file", "uid": 0} 2021-02-12T17:54:33.075 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => (item=/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000) => {"ansible_loop_var": "item", "changed": true, "checksum": "bcdfbb1de91ef82e0e4ed8af725c9ad52593eb79", "dest": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000/docker-mirror.crt", "gid": 0, "group": "root", "item": "/etc/containers/certs.d/docker-mirror.front.sepia.ceph.com:5000", "md5sum": "a3fa0991425692932fcf703c1a9a483e", "mode": "0644", "owner": "root", "secontext": "system_u:object_r:etc_t:s0", "size": 2334, "src": "/home/ubuntu/.ansible/tmp/ansible-tmp-1613152472.6082184-90616059238424/source", "state": "file", "uid": 0} 2021-02-12T17:54:33.112 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:33 +0000 (0:00:01.013) 0:04:46.504 ******* 2021-02-12T17:54:36.277 INFO:teuthology.task.ansible.out: TASK [container-host : Install registries-conf-ctl] **************************** 2021-02-12T17:54:36.279 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["/bin/pip3", "install", "-U", "git+https://github.com/sebastian-philipp/registries-conf-ctl"], "name": ["git+https://github.com/sebastian-philipp/registries-conf-ctl"], "requirements": null, "state": "latest", "stderr": "WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead.\n", "stderr_lines": ["WARNING: Running pip install with root privileges is generally not a good idea. Try `pip3 install --user` instead."], "stdout": "Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl\n Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-1ua4fbgi-build\nCollecting toml (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl\nCollecting docopt (from registries-conf-ctl==1.0)\n Downloading https://files.pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz\nInstalling collected packages: toml, docopt, registries-conf-ctl\n Running setup.py install for docopt: started\n Running setup.py install for docopt: finished with status 'done'\n Running setup.py install for registries-conf-ctl: started\n Running setup.py install for registries-conf-ctl: finished with status 'done'\nSuccessfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2\n", "stdout_lines": ["Collecting git+https://github.com/sebastian-philipp/registries-conf-ctl", " Cloning https://github.com/sebastian-philipp/registries-conf-ctl to ./pip-1ua4fbgi-build", "Collecting toml (from registries-conf-ctl==1.0)", " Downloading https://files.pythonhosted.org/packages/44/6f/7120676b6d73228c96e17f1f794d8ab046fc910d781c8d151120c3f1569e/toml-0.10.2-py2.py3-none-any.whl", "Collecting docopt (from registries-conf-ctl==1.0)", " Downloading https://files. 2021-02-12T17:54:36.280 INFO:teuthology.task.ansible.out:pythonhosted.org/packages/a2/55/8f8cab2afd404cf578136ef2cc5dfb50baa1761b68c9da1fb1e4eed343c9/docopt-0.6.2.tar.gz", "Installing collected packages: toml, docopt, registries-conf-ctl", " Running setup.py install for docopt: started", " Running setup.py install for docopt: finished with status 'done'", " Running setup.py install for registries-conf-ctl: started", " Running setup.py install for registries-conf-ctl: finished with status 'done'", "Successfully installed docopt-0.6.2 registries-conf-ctl-1.0 toml-0.10.2"], "version": null, "virtualenv": null} 2021-02-12T17:54:36.314 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:36 +0000 (0:00:03.201) 0:04:49.705 ******* 2021-02-12T17:54:36.341 INFO:teuthology.task.ansible.out: TASK [container-host : Check for docker's daemon.json] ************************* 2021-02-12T17:54:36.342 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:36.379 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:36 +0000 (0:00:00.064) 0:04:49.770 ******* 2021-02-12T17:54:36.405 INFO:teuthology.task.ansible.out: TASK [container-host : Create /etc/containers/registries.conf if necessary] **** 2021-02-12T17:54:36.406 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:36.441 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:36 +0000 (0:00:00.061) 0:04:49.832 ******* 2021-02-12T17:54:36.798 INFO:teuthology.task.ansible.out: TASK [container-host : Add local docker.io registry mirror] ******************** 2021-02-12T17:54:36.799 INFO:teuthology.task.ansible.out:changed: [smithi107.front.sepia.ceph.com] => {"changed": true, "cmd": ["registries-conf-ctl", "add-mirror", "docker.io", "docker-mirror.front.sepia.ceph.com:5000"], "delta": "0:00:00.169454", "end": "2021-02-12 17:54:36.778453", "rc": 0, "start": "2021-02-12 17:54:36.608999", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} 2021-02-12T17:54:36.836 INFO:teuthology.task.ansible.out:Friday 12 February 2021 17:54:36 +0000 (0:00:00.395) 0:04:50.228 ******* 2021-02-12T17:54:36.862 INFO:teuthology.task.ansible.out: TASK [container-host : Restart docker service] ********************************* 2021-02-12T17:54:36.864 INFO:teuthology.task.ansible.out:skipping: [smithi107.front.sepia.ceph.com] => {"changed": false, "skip_reason": "Conditional result was False"} 2021-02-12T17:54:36.958 INFO:teuthology.task.ansible.out: PLAY [cobbler] ***************************************************************** skipping: no hosts matched 2021-02-12T17:54:36.962 INFO:teuthology.task.ansible.out: PLAY [paddles] ***************************************************************** skipping: no hosts matched 2021-02-12T17:54:36.965 INFO:teuthology.task.ansible.out: PLAY [pulpito] ***************************************************************** skipping: no hosts matched PLAY RECAP ********************************************************************* smithi107.front.sepia.ceph.com : ok=126 changed=46 unreachable=0 failed=0 skipped=184 rescued=0 ignored=3 2021-02-12T17:54:36.966 INFO:teuthology.task.ansible.out: Friday 12 February 2021 17:54:36 +0000 (0:00:00.129) 0:04:50.357 ******* =============================================================================== testnode : Install packages -------------------------------------------- 45.40s common : nrpe - Ensure SELinux policy is up to date -------------------- 38.42s users : Create all admin users with sudo access. ----------------------- 31.39s users : Update authorized_keys using the keys repo --------------------- 28.61s common : nrpe - Remove SELinux policy package -------------------------- 16.82s common : nrpe - Load SELinux policy package ---------------------------- 16.64s users : Remove revoked users ------------------------------------------- 11.46s testnode : Zap all non-root disks --------------------------------------- 8.16s common : Install Kerberos Packages (RedHat) ----------------------------- 7.62s testnode : Make sure rpm dependencies are installed --------------------- 5.38s common : Make sure smartmontools is installed --------------------------- 3.34s container-host : Install registries-conf-ctl ---------------------------- 3.20s common : nrpe - Install SELinux dependencies ---------------------------- 2.75s testnode : Ensure ceph packages are not present. ------------------------ 2.57s testnode : Rebuild rpmdb ------------------------------------------------ 2.25s testnode : Ensure ceph dependency packages are not present. ------------- 2.14s testnode : Create logical volume(s) ------------------------------------- 2.04s testnode : Blow away lingering OSD data and FSIDs ----------------------- 1.91s users : Update authorized_keys for each user with literal keys ---------- 1.80s Gathering Facts --------------------------------------------------------- 1.52s 2021-02-12T17:54:37.245 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi107.front.sepia.ceph.com')] 2021-02-12T17:54:37.246 INFO:teuthology.orchestra.remote:Trying to reconnect to host 2021-02-12T17:54:37.248 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi107.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2021-02-12T17:54:37.319 DEBUG:teuthology.orchestra.run.smithi107:> true 2021-02-12T17:54:37.414 INFO:teuthology.run_tasks:Running task clock... 2021-02-12T17:54:37.434 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2021-02-12T17:54:37.434 INFO:teuthology.orchestra.run:Running command with timeout 360 2021-02-12T17:54:37.435 DEBUG:teuthology.orchestra.run.smithi107:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-12T17:54:37.488 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2021-02-12T17:54:37.503 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2021-02-12T17:54:37.535 INFO:teuthology.orchestra.run.smithi107.stderr:sudo: ntpd: command not found 2021-02-12T17:54:37.546 INFO:teuthology.orchestra.run.smithi107.stdout:506 Cannot talk to daemon 2021-02-12T17:54:37.561 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntp.service: Unit ntp.service not found. 2021-02-12T17:54:37.575 INFO:teuthology.orchestra.run.smithi107.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2021-02-12T17:54:37.635 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2021-02-12T17:54:37.638 INFO:teuthology.orchestra.run.smithi107.stdout:210 Number of sources = 4 2021-02-12T17:54:37.638 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-02-12T17:54:37.639 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2021-02-12T17:54:37.639 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-12T17:54:37.639 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-12T17:54:37.639 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-12T17:54:37.639 INFO:teuthology.orchestra.run.smithi107.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2021-02-12T17:54:37.643 INFO:teuthology.run_tasks:Running task install... 2021-02-12T17:54:37.672 DEBUG:teuthology.task.install:project ceph 2021-02-12T17:54:37.673 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b'}} 2021-02-12T17:54:37.673 DEBUG:teuthology.task.install:config {'sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b'} 2021-02-12T17:54:37.673 INFO:teuthology.task.install:Using flavor: basic 2021-02-12T17:54:37.679 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2021-02-12T17:54:37.679 INFO:teuthology.task.install:extra packages: [] 2021-02-12T17:54:37.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo lsb_release -is 2021-02-12T17:54:37.714 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS 2021-02-12T17:54:37.715 DEBUG:teuthology.misc:System to be installed: CentOS 2021-02-12T17:54:37.715 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': [], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'flavor': 'basic', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b', 'tag': None, 'wait_for_package': False} 2021-02-12T17:54:37.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo lsb_release -is 2021-02-12T17:54:37.787 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS 2021-02-12T17:54:37.789 DEBUG:teuthology.misc:System to be installed: CentOS 2021-02-12T17:54:37.789 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5a3430b93fcfcd31d38534758b8997c894688c4b 2021-02-12T17:54:37.961 INFO:teuthology.task.install.rpm:Pulling from https://1.chacra.ceph.com/r/ceph/wip-dmclock-fix-high-client-profile/5a3430b93fcfcd31d38534758b8997c894688c4b/centos/8/flavors/default/ 2021-02-12T17:54:38.368 INFO:teuthology.task.install.rpm:Package version is 17.0.0-713.g5a3430b9 2021-02-12T17:54:38.654 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://1.chacra.ceph.com/r/ceph/wip-dmclock-fix-high-client-profile/5a3430b93fcfcd31d38534758b8997c894688c4b/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dmclock-fix-high-client-profile/5a3430b93fcfcd31d38534758b8997c894688c4b/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://1.chacra.ceph.com/r/ceph/wip-dmclock-fix-high-client-profile/5a3430b93fcfcd31d38534758b8997c894688c4b/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2021-02-12T17:54:38.655 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:54:38.655 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/yum.repos.d/ceph.repo 2021-02-12T17:54:38.683 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2021-02-12T17:54:38.684 DEBUG:teuthology.orchestra.run.smithi107:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;sha1/5a3430b93fcfcd31d38534758b8997c894688c4b/;g' /etc/yum.repos.d/ceph.repo ; fi 2021-02-12T17:54:38.753 DEBUG:teuthology.orchestra.run.smithi107:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2021-02-12T17:54:38.831 DEBUG:teuthology.orchestra.run.smithi107:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2021-02-12T17:54:38.898 INFO:teuthology.orchestra.run.smithi107.stdout:check_obsoletes = 1 2021-02-12T17:54:38.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum clean all 2021-02-12T17:54:39.256 INFO:teuthology.orchestra.run.smithi107.stdout:107 files removed 2021-02-12T17:54:39.285 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-radosgw 2021-02-12T17:54:40.398 INFO:teuthology.orchestra.run.smithi107.stdout:ceph packages for x86_64 99 kB/s | 73 kB 00:00 2021-02-12T17:54:41.059 INFO:teuthology.orchestra.run.smithi107.stdout:ceph noarch packages 24 kB/s | 15 kB 00:00 2021-02-12T17:54:41.733 INFO:teuthology.orchestra.run.smithi107.stdout:ceph source packages 2.2 kB/s | 1.4 kB 00:00 2021-02-12T17:54:41.891 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - AppStream 47 MB/s | 6.3 MB 00:00 2021-02-12T17:54:42.968 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - Base 26 MB/s | 2.3 MB 00:00 2021-02-12T17:54:43.417 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS-8 - Extras 114 kB/s | 8.6 kB 00:00 2021-02-12T17:54:43.847 INFO:teuthology.orchestra.run.smithi107.stdout:Extra Packages for Enterprise Linux 23 MB/s | 8.9 MB 00:00 2021-02-12T17:54:45.724 INFO:teuthology.orchestra.run.smithi107.stdout:lab-extras 4.4 MB/s | 24 kB 00:00 2021-02-12T17:54:46.852 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 13 M 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrading: 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: librados2 x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 3.7 M 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1 x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 3.3 M 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:54:46.868 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 6.0 M 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 24 M 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 25 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs x86_64 2.8-1.el8 epel 306 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2 x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 798 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq x86_64 0.9.0-2.el8 CentOS-Base 47 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1 x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 486 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka x86_64 0.11.4-1.el8 CentOS-AppStream 353 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2 x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 4.0 M 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust x86_64 2.8.1-11.el8 CentOS-AppStream 259 k 2021-02-12T17:54:46.869 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 44 k 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 77 k 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 186 k 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable noarch 0.7.2-14.el8 CentOS-AppStream 44 k 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 387 k 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 368 k 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 114 k 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout:Install 22 Packages 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout:Upgrade 2 Packages 2021-02-12T17:54:46.870 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:54:46.871 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 57 M 2021-02-12T17:54:46.871 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:54:47.841 INFO:teuthology.orchestra.run.smithi107.stdout:(1/24): ceph-base-17.0.0-713.g5a3430b9.el8.x86_ 6.2 MB/s | 6.0 MB 00:00 2021-02-12T17:54:47.898 INFO:teuthology.orchestra.run.smithi107.stdout:(2/24): ceph-selinux-17.0.0-713.g5a3430b9.el8.x 433 kB/s | 25 kB 00:00 2021-02-12T17:54:48.064 INFO:teuthology.orchestra.run.smithi107.stdout:(3/24): libcephfs2-17.0.0-713.g5a3430b9.el8.x86 4.7 MB/s | 798 kB 00:00 2021-02-12T17:54:48.201 INFO:teuthology.orchestra.run.smithi107.stdout:(4/24): libradosstriper1-17.0.0-713.g5a3430b9.e 3.5 MB/s | 486 kB 00:00 2021-02-12T17:54:48.791 INFO:teuthology.orchestra.run.smithi107.stdout:(5/24): ceph-radosgw-17.0.0-713.g5a3430b9.el8.x 6.5 MB/s | 13 MB 00:01 2021-02-12T17:54:48.802 INFO:teuthology.orchestra.run.smithi107.stdout:(6/24): librgw2-17.0.0-713.g5a3430b9.el8.x86_64 6.7 MB/s | 4.0 MB 00:00 2021-02-12T17:54:48.814 INFO:teuthology.orchestra.run.smithi107.stdout:(7/24): python3-ceph-argparse-17.0.0-713.g5a343 1.8 MB/s | 44 kB 00:00 2021-02-12T17:54:48.827 INFO:teuthology.orchestra.run.smithi107.stdout:(8/24): python3-ceph-common-17.0.0-713.g5a3430b 2.9 MB/s | 77 kB 00:00 2021-02-12T17:54:48.840 INFO:teuthology.orchestra.run.smithi107.stdout:(9/24): python3-cephfs-17.0.0-713.g5a3430b9.el8 7.3 MB/s | 186 kB 00:00 2021-02-12T17:54:48.870 INFO:teuthology.orchestra.run.smithi107.stdout:(10/24): python3-rados-17.0.0-713.g5a3430b9.el8 8.9 MB/s | 387 kB 00:00 2021-02-12T17:54:48.909 INFO:teuthology.orchestra.run.smithi107.stdout:(11/24): python3-rgw-17.0.0-713.g5a3430b9.el8.x 2.9 MB/s | 114 kB 00:00 2021-02-12T17:54:48.917 INFO:teuthology.orchestra.run.smithi107.stdout:(12/24): python3-rbd-17.0.0-713.g5a3430b9.el8.x 4.6 MB/s | 368 kB 00:00 2021-02-12T17:54:48.942 INFO:teuthology.orchestra.run.smithi107.stdout:(13/24): librdkafka-0.11.4-1.el8.x86_64.rpm 10 MB/s | 353 kB 00:00 2021-02-12T17:54:48.944 INFO:teuthology.orchestra.run.smithi107.stdout:(14/24): python3-prettytable-0.7.2-14.el8.noarc 33 MB/s | 44 kB 00:00 2021-02-12T17:54:48.945 INFO:teuthology.orchestra.run.smithi107.stdout:(15/24): librabbitmq-0.9.0-2.el8.x86_64.rpm 36 MB/s | 47 kB 00:00 2021-02-12T17:54:48.950 INFO:teuthology.orchestra.run.smithi107.stdout:(16/24): lttng-ust-2.8.1-11.el8.x86_64.rpm 7.8 MB/s | 259 kB 00:00 2021-02-12T17:54:49.180 INFO:teuthology.orchestra.run.smithi107.stdout:(17/24): ceph-common-17.0.0-713.g5a3430b9.el8.x 10 MB/s | 24 MB 00:02 2021-02-12T17:54:49.181 INFO:teuthology.orchestra.run.smithi107.stdout:(18/24): fmt-6.2.1-1.el8.x86_64.rpm 494 kB/s | 116 kB 00:00 2021-02-12T17:54:49.198 INFO:teuthology.orchestra.run.smithi107.stdout:(19/24): liboath-2.6.2-3.el8.x86_64.rpm 3.5 MB/s | 59 kB 00:00 2021-02-12T17:54:49.201 INFO:teuthology.orchestra.run.smithi107.stdout:(20/24): gperftools-libs-2.8-1.el8.x86_64.rpm 1.2 MB/s | 306 kB 00:00 2021-02-12T17:54:49.214 INFO:teuthology.orchestra.run.smithi107.stdout:(21/24): libunwind-1.3.1-3.el8.x86_64.rpm 4.4 MB/s | 75 kB 00:00 2021-02-12T17:54:49.277 INFO:teuthology.orchestra.run.smithi107.stdout:(22/24): leveldb-1.22-1.el8.x86_64.rpm 1.8 MB/s | 181 kB 00:00 2021-02-12T17:54:49.420 INFO:teuthology.orchestra.run.smithi107.stdout:(23/24): librbd1-17.0.0-713.g5a3430b9.el8.x86_6 16 MB/s | 3.3 MB 00:00 2021-02-12T17:54:49.503 INFO:teuthology.orchestra.run.smithi107.stdout:(24/24): librados2-17.0.0-713.g5a3430b9.el8.x86 12 MB/s | 3.7 MB 00:00 2021-02-12T17:54:49.506 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:54:49.506 INFO:teuthology.orchestra.run.smithi107.stdout:Total 22 MB/s | 57 MB 00:02 2021-02-12T17:54:49.510 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:54:49.554 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:54:49.554 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:54:49.888 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:54:49.890 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:54:51.044 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:54:51.212 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/26 2021-02-12T17:54:51.391 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/26 2021-02-12T17:54:51.598 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-02-12T17:54:52.001 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/26 2021-02-12T17:54:52.190 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librados2-2:17.0.0-713.g5a3430b9.el8.x86_64 4/26 2021-02-12T17:54:52.291 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-2:17.0.0-713.g5a3430b9.el8.x86_64 4/26 2021-02-12T17:54:52.513 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rados-2:17.0.0-713.g5a3430b9.el8.x86_64 5/26 2021-02-12T17:54:52.681 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs2-2:17.0.0-713.g5a3430b9.el8.x86_64 6/26 2021-02-12T17:54:53.026 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs2-2:17.0.0-713.g5a3430b9.el8.x86_64 6/26 2021-02-12T17:54:53.223 INFO:teuthology.orchestra.run.smithi107.stdout: Upgrading : librbd1-2:17.0.0-713.g5a3430b9.el8.x86_64 7/26 2021-02-12T17:54:53.271 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-2:17.0.0-713.g5a3430b9.el8.x86_64 7/26 2021-02-12T17:54:53.413 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librabbitmq-0.9.0-2.el8.x86_64 8/26 2021-02-12T17:54:53.556 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librdkafka-0.11.4-1.el8.x86_64 9/26 2021-02-12T17:54:54.000 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librdkafka-0.11.4-1.el8.x86_64 9/26 2021-02-12T17:54:54.173 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librgw2-2:17.0.0-713.g5a3430b9.el8.x86_64 10/26 2021-02-12T17:54:54.235 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librgw2-2:17.0.0-713.g5a3430b9.el8.x86_64 10/26 2021-02-12T17:54:54.403 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : leveldb-1.22-1.el8.x86_64 11/26 2021-02-12T17:54:54.540 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-argparse-2:17.0.0-713.g5a3430b9.el8.x 12/26 2021-02-12T17:54:54.717 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cephfs-2:17.0.0-713.g5a3430b9.el8.x86_64 13/26 2021-02-12T17:54:54.881 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rgw-2:17.0.0-713.g5a3430b9.el8.x86_64 14/26 2021-02-12T17:54:55.082 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rbd-2:17.0.0-713.g5a3430b9.el8.x86_64 15/26 2021-02-12T17:54:55.239 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libradosstriper1-2:17.0.0-713.g5a3430b9.el8.x86_64 16/26 2021-02-12T17:54:55.312 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libradosstriper1-2:17.0.0-713.g5a3430b9.el8.x86_64 16/26 2021-02-12T17:54:55.516 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 17/26 2021-02-12T17:54:55.723 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : gperftools-libs-2.8-1.el8.x86_64 18/26 2021-02-12T17:54:55.914 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 19/26 2021-02-12T17:54:56.064 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-ceph-common-2:17.0.0-713.g5a3430b9.el8.x86 20/26 2021-02-12T17:54:59.623 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:17.0.0-713.g5a3430b9.el8.x86_64 21/26 2021-02-12T17:54:59.822 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-common-2:17.0.0-713.g5a3430b9.el8.x86_64 21/26 2021-02-12T17:55:00.520 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-common-2:17.0.0-713.g5a3430b9.el8.x86_64 21/26 2021-02-12T17:55:00.806 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-base-2:17.0.0-713.g5a3430b9.el8.x86_64 22/26 2021-02-12T17:55:00.902 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-base-2:17.0.0-713.g5a3430b9.el8.x86_64 22/26 2021-02-12T17:55:01.046 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-selinux-2:17.0.0-713.g5a3430b9.el8.x86_64 23/26 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-selinux-2:17.0.0-713.g5a3430b9.el8.x86_64 23/26 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /sys 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /proc 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /run 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /mnt 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /var/tmp 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /home 2021-02-12T17:55:17.783 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /tmp 2021-02-12T17:55:17.784 INFO:teuthology.orchestra.run.smithi107.stdout:skipping the directory /dev 2021-02-12T17:55:17.784 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:19.052 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-radosgw-2:17.0.0-713.g5a3430b9.el8.x86_64 24/26 2021-02-12T17:55:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-radosgw-2:17.0.0-713.g5a3430b9.el8.x86_64 24/26 2021-02-12T17:55:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-02-12T17:55:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 25/26 2021-02-12T17:55:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 26/26 2021-02-12T17:55:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 26/26 2021-02-12T17:55:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-base-2:17.0.0-713.g5a3430b9.el8.x86_64 1/26 2021-02-12T17:55:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-common-2:17.0.0-713.g5a3430b9.el8.x86_64 2/26 2021-02-12T17:55:21.271 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-radosgw-2:17.0.0-713.g5a3430b9.el8.x86_64 3/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-selinux-2:17.0.0-713.g5a3430b9.el8.x86_64 4/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs2-2:17.0.0-713.g5a3430b9.el8.x86_64 5/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libradosstriper1-2:17.0.0-713.g5a3430b9.el8.x86_64 6/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librgw2-2:17.0.0-713.g5a3430b9.el8.x86_64 7/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-argparse-2:17.0.0-713.g5a3430b9.el8.x 8/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-ceph-common-2:17.0.0-713.g5a3430b9.el8.x86 9/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cephfs-2:17.0.0-713.g5a3430b9.el8.x86_64 10/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rados-2:17.0.0-713.g5a3430b9.el8.x86_64 11/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rbd-2:17.0.0-713.g5a3430b9.el8.x86_64 12/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rgw-2:17.0.0-713.g5a3430b9.el8.x86_64 13/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librdkafka-0.11.4-1.el8.x86_64 14/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/26 2021-02-12T17:55:21.272 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librabbitmq-0.9.0-2.el8.x86_64 17/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 18/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : gperftools-libs-2.8-1.el8.x86_64 19/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : leveldb-1.22-1.el8.x86_64 20/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 21/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 22/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-2:17.0.0-713.g5a3430b9.el8.x86_64 23/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 24/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-2:17.0.0-713.g5a3430b9.el8.x86_64 25/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 26/26 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout:Upgraded: 2021-02-12T17:55:21.273 INFO:teuthology.orchestra.run.smithi107.stdout: librados2-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: librbd1-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-base-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-common-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-radosgw-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-selinux-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: fmt-6.2.1-1.el8.x86_64 2021-02-12T17:55:21.274 INFO:teuthology.orchestra.run.smithi107.stdout: gperftools-libs-2.8-1.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: leveldb-1.22-1.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs2-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: liboath-2.6.2-3.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: librabbitmq-0.9.0-2.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: libradosstriper1-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: librdkafka-0.11.4-1.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: librgw2-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: libunwind-1.3.1-3.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: lttng-ust-2.8.1-11.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-argparse-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: python3-ceph-common-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cephfs-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.275 INFO:teuthology.orchestra.run.smithi107.stdout: python3-prettytable-0.7.2-14.el8.noarch 2021-02-12T17:55:21.276 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rados-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.276 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rbd-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.276 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rgw-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:21.276 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:21.276 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:55:21.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-test 2021-02-12T17:55:21.843 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:36 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:55:22.292 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 52 M 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout: jq x86_64 1.5-12.el8 CentOS-AppStream 161 k 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout: libxslt x86_64 1.1.32-5.el8 CentOS-Base 250 k 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout: oniguruma x86_64 6.8.2-2.el8 CentOS-AppStream 187 k 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout: socat x86_64 1.7.3.3-2.el8 CentOS-AppStream 302 k 2021-02-12T17:55:22.295 INFO:teuthology.orchestra.run.smithi107.stdout: xmlstarlet x86_64 1.6.1-11.el8 epel 69 k 2021-02-12T17:55:22.296 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:22.296 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:55:22.296 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:22.296 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2021-02-12T17:55:22.296 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:22.296 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 53 M 2021-02-12T17:55:22.296 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 228 M 2021-02-12T17:55:22.296 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:55:22.331 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): jq-1.5-12.el8.x86_64.rpm 4.6 MB/s | 161 kB 00:00 2021-02-12T17:55:22.334 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 4.9 MB/s | 187 kB 00:00 2021-02-12T17:55:22.336 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): socat-1.7.3.3-2.el8.x86_64.rpm 54 MB/s | 302 kB 00:00 2021-02-12T17:55:22.338 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): libxslt-1.1.32-5.el8.x86_64.rpm 60 MB/s | 250 kB 00:00 2021-02-12T17:55:22.415 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): xmlstarlet-1.6.1-11.el8.x86_64.rpm 882 kB/s | 69 kB 00:00 2021-02-12T17:55:24.850 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-test-17.0.0-713.g5a3430b9.el8.x86_6 20 MB/s | 52 MB 00:02 2021-02-12T17:55:24.853 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:55:24.854 INFO:teuthology.orchestra.run.smithi107.stdout:Total 21 MB/s | 53 MB 00:02 2021-02-12T17:55:24.858 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:55:24.882 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:55:24.882 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:55:25.173 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:55:25.221 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:55:25.958 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:55:26.192 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libxslt-1.1.32-5.el8.x86_64 1/6 2021-02-12T17:55:26.385 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : xmlstarlet-1.6.1-11.el8.x86_64 2/6 2021-02-12T17:55:26.593 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : socat-1.7.3.3-2.el8.x86_64 3/6 2021-02-12T17:55:26.774 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 4/6 2021-02-12T17:55:26.839 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 4/6 2021-02-12T17:55:31.709 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : jq-1.5-12.el8.x86_64 5/6 2021-02-12T17:55:31.946 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-test-2:17.0.0-713.g5a3430b9.el8.x86_64 6/6 2021-02-12T17:55:32.181 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-test-2:17.0.0-713.g5a3430b9.el8.x86_64 6/6 2021-02-12T17:55:32.181 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-test-2:17.0.0-713.g5a3430b9.el8.x86_64 1/6 2021-02-12T17:55:32.181 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : jq-1.5-12.el8.x86_64 2/6 2021-02-12T17:55:32.181 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 3/6 2021-02-12T17:55:32.181 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : socat-1.7.3.3-2.el8.x86_64 4/6 2021-02-12T17:55:32.181 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libxslt-1.1.32-5.el8.x86_64 5/6 2021-02-12T17:55:32.552 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : xmlstarlet-1.6.1-11.el8.x86_64 6/6 2021-02-12T17:55:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:32.553 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:55:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-test-2:17.0.0-713.g5a3430b9.el8.x86_64 jq-1.5-12.el8.x86_64 2021-02-12T17:55:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: libxslt-1.1.32-5.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2021-02-12T17:55:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: socat-1.7.3.3-2.el8.x86_64 xmlstarlet-1.6.1-11.el8.x86_64 2021-02-12T17:55:32.553 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:32.553 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:55:32.626 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph 2021-02-12T17:55:33.074 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:00:48 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:55:33.591 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:55:33.595 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: ceph x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 6.4 k 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 2.4 M 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 1.5 M 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core noarch 2:17.0.0-713.g5a3430b9.el8 ceph-noarch 222 k 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 5.0 M 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 18 M 2021-02-12T17:55:33.596 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-chardet noarch 3.0.4-7.el8 CentOS-Base 195 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-lxml x86_64 4.2.3-1.el8 CentOS-AppStream 1.5 M 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako noarch 1.0.6-13.el8 CentOS-AppStream 157 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe x86_64 0.23-19.el8 CentOS-AppStream 39 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2021-02-12T17:55:33.597 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pysocks noarch 1.6.8-3.el8 CentOS-Base 34 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz noarch 2017.2-9.el8 CentOS-AppStream 54 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests noarch 2.20.0-2.1.el8_1 CentOS-Base 123 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3 noarch 1.24.2-4.el8 CentOS-Base 176 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress noarch 1.2.1-2.el8.1 epel 241 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webencodings noarch 0.5.1-6.el8 CentOS-AppStream 27 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug noarch 0.12.2-4.el8 CentOS-AppStream 457 k 2021-02-12T17:55:33.598 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2021-02-12T17:55:33.599 INFO:teuthology.orchestra.run.smithi107.stdout:Installing weak dependencies: 2021-02-12T17:55:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cssselect noarch 0.9.2-10.el8 epel 40 k 2021-02-12T17:55:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: python3-html5lib noarch 1:0.999999999-6.el8 CentOS-AppStream 214 k 2021-02-12T17:55:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:33.599 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:55:33.599 INFO:teuthology.orchestra.run.smithi107.stdout:====================================================================================== 2021-02-12T17:55:33.599 INFO:teuthology.orchestra.run.smithi107.stdout:Install 36 Packages 2021-02-12T17:55:33.599 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:33.600 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 32 M 2021-02-12T17:55:33.600 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 114 M 2021-02-12T17:55:33.600 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:55:33.712 INFO:teuthology.orchestra.run.smithi107.stdout:(1/36): ceph-17.0.0-713.g5a3430b9.el8.x86_64.rp 58 kB/s | 6.4 kB 00:00 2021-02-12T17:55:33.936 INFO:teuthology.orchestra.run.smithi107.stdout:(2/36): ceph-mds-17.0.0-713.g5a3430b9.el8.x86_6 7.2 MB/s | 2.4 MB 00:00 2021-02-12T17:55:33.941 INFO:teuthology.orchestra.run.smithi107.stdout:(3/36): ceph-mgr-17.0.0-713.g5a3430b9.el8.x86_6 4.5 MB/s | 1.5 MB 00:00 2021-02-12T17:55:34.002 INFO:teuthology.orchestra.run.smithi107.stdout:(4/36): ceph-mgr-modules-core-17.0.0-713.g5a343 3.6 MB/s | 222 kB 00:00 2021-02-12T17:55:34.034 INFO:teuthology.orchestra.run.smithi107.stdout:(5/36): python3-html5lib-0.999999999-6.el8.noar 6.5 MB/s | 214 kB 00:00 2021-02-12T17:55:34.052 INFO:teuthology.orchestra.run.smithi107.stdout:(6/36): python3-lxml-4.2.3-1.el8.x86_64.rpm 83 MB/s | 1.5 MB 00:00 2021-02-12T17:55:34.054 INFO:teuthology.orchestra.run.smithi107.stdout:(7/36): python3-mako-1.0.6-13.el8.noarch.rpm 61 MB/s | 157 kB 00:00 2021-02-12T17:55:34.056 INFO:teuthology.orchestra.run.smithi107.stdout:(8/36): python3-markupsafe-0.23-19.el8.x86_64.r 34 MB/s | 39 kB 00:00 2021-02-12T17:55:34.058 INFO:teuthology.orchestra.run.smithi107.stdout:(9/36): python3-pytz-2017.2-9.el8.noarch.rpm 36 MB/s | 54 kB 00:00 2021-02-12T17:55:34.059 INFO:teuthology.orchestra.run.smithi107.stdout:(10/36): python3-webencodings-0.5.1-6.el8.noarc 22 MB/s | 27 kB 00:00 2021-02-12T17:55:34.065 INFO:teuthology.orchestra.run.smithi107.stdout:(11/36): python3-werkzeug-0.12.2-4.el8.noarch.r 76 MB/s | 457 kB 00:00 2021-02-12T17:55:34.068 INFO:teuthology.orchestra.run.smithi107.stdout:(12/36): python3-chardet-3.0.4-7.el8.noarch.rpm 67 MB/s | 195 kB 00:00 2021-02-12T17:55:34.069 INFO:teuthology.orchestra.run.smithi107.stdout:(13/36): python3-pysocks-1.6.8-3.el8.noarch.rpm 31 MB/s | 34 kB 00:00 2021-02-12T17:55:34.072 INFO:teuthology.orchestra.run.smithi107.stdout:(14/36): python3-requests-2.20.0-2.1.el8_1.noar 59 MB/s | 123 kB 00:00 2021-02-12T17:55:34.075 INFO:teuthology.orchestra.run.smithi107.stdout:(15/36): python3-urllib3-1.24.2-4.el8.noarch.rp 64 MB/s | 176 kB 00:00 2021-02-12T17:55:34.139 INFO:teuthology.orchestra.run.smithi107.stdout:(16/36): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 675 kB/s | 44 kB 00:00 2021-02-12T17:55:34.187 INFO:teuthology.orchestra.run.smithi107.stdout:(17/36): python3-beautifulsoup4-4.6.3-2.el8.1.n 3.8 MB/s | 185 kB 00:00 2021-02-12T17:55:34.205 INFO:teuthology.orchestra.run.smithi107.stdout:(18/36): python3-cheroot-8.5.2-1.el8.noarch.rpm 9.7 MB/s | 173 kB 00:00 2021-02-12T17:55:34.239 INFO:teuthology.orchestra.run.smithi107.stdout:(19/36): python3-cherrypy-18.4.0-1.el8.noarch.r 11 MB/s | 384 kB 00:00 2021-02-12T17:55:34.255 INFO:teuthology.orchestra.run.smithi107.stdout:(20/36): python3-cssselect-0.9.2-10.el8.noarch. 2.4 MB/s | 40 kB 00:00 2021-02-12T17:55:34.272 INFO:teuthology.orchestra.run.smithi107.stdout:(21/36): python3-jaraco-6.2-6.el8.noarch.rpm 686 kB/s | 11 kB 00:00 2021-02-12T17:55:34.288 INFO:teuthology.orchestra.run.smithi107.stdout:(22/36): python3-jaraco-functools-2.0-4.el8.noa 1.0 MB/s | 18 kB 00:00 2021-02-12T17:55:34.305 INFO:teuthology.orchestra.run.smithi107.stdout:(23/36): python3-logutils-0.3.5-11.el8.noarch.r 2.8 MB/s | 49 kB 00:00 2021-02-12T17:55:34.322 INFO:teuthology.orchestra.run.smithi107.stdout:(24/36): python3-more-itertools-7.2.0-3.el8.noa 3.5 MB/s | 59 kB 00:00 2021-02-12T17:55:34.340 INFO:teuthology.orchestra.run.smithi107.stdout:(25/36): python3-pecan-1.3.2-9.el8.noarch.rpm 15 MB/s | 283 kB 00:00 2021-02-12T17:55:34.357 INFO:teuthology.orchestra.run.smithi107.stdout:(26/36): python3-portend-2.6-1.el8.noarch.rpm 942 kB/s | 16 kB 00:00 2021-02-12T17:55:34.374 INFO:teuthology.orchestra.run.smithi107.stdout:(27/36): python3-simplegeneric-0.8.1-17.el8.noa 1.1 MB/s | 19 kB 00:00 2021-02-12T17:55:34.391 INFO:teuthology.orchestra.run.smithi107.stdout:(28/36): python3-singledispatch-3.4.0.3-18.el8. 1.4 MB/s | 24 kB 00:00 2021-02-12T17:55:34.408 INFO:teuthology.orchestra.run.smithi107.stdout:(29/36): python3-tempora-1.14.1-5.el8.noarch.rp 1.7 MB/s | 29 kB 00:00 2021-02-12T17:55:34.428 INFO:teuthology.orchestra.run.smithi107.stdout:(30/36): python3-trustme-0.6.0-4.el8.noarch.rpm 1.4 MB/s | 27 kB 00:00 2021-02-12T17:55:34.446 INFO:teuthology.orchestra.run.smithi107.stdout:(31/36): python3-waitress-1.2.1-2.el8.1.noarch. 13 MB/s | 241 kB 00:00 2021-02-12T17:55:34.465 INFO:teuthology.orchestra.run.smithi107.stdout:(32/36): python3-webob-1.8.5-1.el8.1.noarch.rpm 14 MB/s | 251 kB 00:00 2021-02-12T17:55:34.482 INFO:teuthology.orchestra.run.smithi107.stdout:(33/36): python3-webtest-2.0.33-1.el8.noarch.rp 4.9 MB/s | 85 kB 00:00 2021-02-12T17:55:34.498 INFO:teuthology.orchestra.run.smithi107.stdout:(34/36): python3-zc-lockfile-2.0-2.el8.noarch.r 1.4 MB/s | 23 kB 00:00 2021-02-12T17:55:34.692 INFO:teuthology.orchestra.run.smithi107.stdout:(35/36): ceph-mon-17.0.0-713.g5a3430b9.el8.x86_ 5.1 MB/s | 5.0 MB 00:00 2021-02-12T17:55:34.831 INFO:teuthology.orchestra.run.smithi107.stdout:(36/36): ceph-osd-17.0.0-713.g5a3430b9.el8.x86_ 20 MB/s | 18 MB 00:00 2021-02-12T17:55:34.834 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:55:34.834 INFO:teuthology.orchestra.run.smithi107.stdout:Total 26 MB/s | 32 MB 00:01 2021-02-12T17:55:34.843 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:55:34.884 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:55:34.884 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:55:35.132 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:55:35.134 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:55:36.474 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:55:36.720 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/36 2021-02-12T17:55:36.930 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/36 2021-02-12T17:55:37.162 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/36 2021-02-12T17:55:37.371 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-waitress-1.2.1-2.el8.1.noarch 4/36 2021-02-12T17:55:37.545 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 5/36 2021-02-12T17:55:37.709 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 6/36 2021-02-12T17:55:37.865 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 7/36 2021-02-12T17:55:38.083 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 8/36 2021-02-12T17:55:38.210 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 9/36 2021-02-12T17:55:38.433 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 10/36 2021-02-12T17:55:38.682 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 11/36 2021-02-12T17:55:38.881 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cssselect-0.9.2-10.el8.noarch 12/36 2021-02-12T17:55:39.070 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 13/36 2021-02-12T17:55:39.311 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pysocks-1.6.8-3.el8.noarch 14/36 2021-02-12T17:55:39.634 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-urllib3-1.24.2-4.el8.noarch 15/36 2021-02-12T17:55:39.843 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-chardet-3.0.4-7.el8.noarch 16/36 2021-02-12T17:55:40.116 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-2.20.0-2.1.el8_1.noarch 17/36 2021-02-12T17:55:40.290 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 18/36 2021-02-12T17:55:40.515 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webencodings-0.5.1-6.el8.noarch 19/36 2021-02-12T17:55:40.725 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-html5lib-1:0.999999999-6.el8.noarch 20/36 2021-02-12T17:55:41.071 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 21/36 2021-02-12T17:55:41.322 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-lxml-4.2.3-1.el8.x86_64 22/36 2021-02-12T17:55:41.510 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 23/36 2021-02-12T17:55:41.640 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pytz-2017.2-9.el8.noarch 24/36 2021-02-12T17:55:41.838 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 25/36 2021-02-12T17:55:42.087 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-portend-2.6-1.el8.noarch 26/36 2021-02-12T17:55:42.299 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 27/36 2021-02-12T17:55:42.508 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 28/36 2021-02-12T17:55:42.876 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-mako-1.0.6-13.el8.noarch 29/36 2021-02-12T17:55:43.206 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 30/36 2021-02-12T17:55:43.606 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-modules-core-2:17.0.0-713.g5a3430b9.el8.n 31/36 2021-02-12T17:55:43.821 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-2:17.0.0-713.g5a3430b9.el8.x86_64 32/36 2021-02-12T17:55:45.344 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-2:17.0.0-713.g5a3430b9.el8.x86_64 32/36 2021-02-12T17:55:45.596 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-osd-2:17.0.0-713.g5a3430b9.el8.x86_64 33/36 2021-02-12T17:55:46.031 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-osd-2:17.0.0-713.g5a3430b9.el8.x86_64 33/36 2021-02-12T17:55:46.237 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mon-2:17.0.0-713.g5a3430b9.el8.x86_64 34/36 2021-02-12T17:55:46.462 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mon-2:17.0.0-713.g5a3430b9.el8.x86_64 34/36 2021-02-12T17:55:46.645 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mds-2:17.0.0-713.g5a3430b9.el8.x86_64 35/36 2021-02-12T17:55:46.665 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mds-2:17.0.0-713.g5a3430b9.el8.x86_64 35/36 2021-02-12T17:55:47.071 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-2:17.0.0-713.g5a3430b9.el8.x86_64 36/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-2:17.0.0-713.g5a3430b9.el8.x86_64 36/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-2:17.0.0-713.g5a3430b9.el8.x86_64 1/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mds-2:17.0.0-713.g5a3430b9.el8.x86_64 2/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-2:17.0.0-713.g5a3430b9.el8.x86_64 3/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mon-2:17.0.0-713.g5a3430b9.el8.x86_64 4/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-osd-2:17.0.0-713.g5a3430b9.el8.x86_64 5/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-modules-core-2:17.0.0-713.g5a3430b9.el8.n 6/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-html5lib-1:0.999999999-6.el8.noarch 7/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-lxml-4.2.3-1.el8.x86_64 8/36 2021-02-12T17:55:47.509 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-mako-1.0.6-13.el8.noarch 9/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 10/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pytz-2017.2-9.el8.noarch 11/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webencodings-0.5.1-6.el8.noarch 12/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 13/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-chardet-3.0.4-7.el8.noarch 14/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pysocks-1.6.8-3.el8.noarch 15/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-2.20.0-2.1.el8_1.noarch 16/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-urllib3-1.24.2-4.el8.noarch 17/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 19/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 20/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 21/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cssselect-0.9.2-10.el8.noarch 22/36 2021-02-12T17:55:47.510 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 23/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 24/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 25/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 26/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 27/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-portend-2.6-1.el8.noarch 28/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 29/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 30/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 31/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 32/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-waitress-1.2.1-2.el8.1.noarch 33/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 34/36 2021-02-12T17:55:47.511 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 35/36 2021-02-12T17:55:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 36/36 2021-02-12T17:55:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:48.341 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:55:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mds-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-modules-core-2:17.0.0-713.g5a3430b9.el8.noarch 2021-02-12T17:55:48.341 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mon-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-osd-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-chardet-3.0.4-7.el8.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cheroot-8.5.2-1.el8.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cssselect-0.9.2-10.el8.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-html5lib-1:0.999999999-6.el8.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-6.2-6.el8.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-logutils-0.3.5-11.el8.noarch 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-lxml-4.2.3-1.el8.x86_64 2021-02-12T17:55:48.342 INFO:teuthology.orchestra.run.smithi107.stdout: python3-mako-1.0.6-13.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-markupsafe-0.23-19.el8.x86_64 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pecan-1.3.2-9.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-portend-2.6-1.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pysocks-1.6.8-3.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pytz-2017.2-9.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-2.20.0-2.1.el8_1.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-tempora-1.14.1-5.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-trustme-0.6.0-4.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-urllib3-1.24.2-4.el8.noarch 2021-02-12T17:55:48.343 INFO:teuthology.orchestra.run.smithi107.stdout: python3-waitress-1.2.1-2.el8.1.noarch 2021-02-12T17:55:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webencodings-0.5.1-6.el8.noarch 2021-02-12T17:55:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webob-1.8.5-1.el8.1.noarch 2021-02-12T17:55:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: python3-webtest-2.0.33-1.el8.noarch 2021-02-12T17:55:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2021-02-12T17:55:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2021-02-12T17:55:48.344 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:48.344 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:55:48.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-base 2021-02-12T17:55:49.067 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:04 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:55:49.416 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-base-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:55:49.528 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:55:49.529 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:55:49.529 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:55:49.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install cephadm 2021-02-12T17:55:50.215 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:05 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:55:50.652 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm noarch 2:17.0.0-713.g5a3430b9.el8 ceph-noarch 69 k 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2021-02-12T17:55:50.653 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:50.654 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 69 k 2021-02-12T17:55:50.654 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 277 k 2021-02-12T17:55:50.654 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:55:50.804 INFO:teuthology.orchestra.run.smithi107.stdout:cephadm-17.0.0-713.g5a3430b9.el8.noarch.rpm 459 kB/s | 69 kB 00:00 2021-02-12T17:55:50.804 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:55:50.805 INFO:teuthology.orchestra.run.smithi107.stdout:Total 455 kB/s | 69 kB 00:00 2021-02-12T17:55:50.805 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:55:50.813 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:55:50.813 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:55:50.815 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:55:50.817 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:55:50.887 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:55:52.726 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 1/1 2021-02-12T17:55:53.032 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 1/1 2021-02-12T17:55:53.208 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 1/1 2021-02-12T17:55:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 1/1 2021-02-12T17:55:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:53.514 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:55:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 2021-02-12T17:55:53.514 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:53.514 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:55:53.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-immutable-object-cache 2021-02-12T17:55:54.150 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:09 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:55:54.656 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 163 k 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2021-02-12T17:55:54.657 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:54.658 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 163 k 2021-02-12T17:55:54.658 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 587 k 2021-02-12T17:55:54.658 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:55:54.828 INFO:teuthology.orchestra.run.smithi107.stdout:ceph-immutable-object-cache-17.0.0-713.g5a3430b 961 kB/s | 163 kB 00:00 2021-02-12T17:55:54.828 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:55:54.829 INFO:teuthology.orchestra.run.smithi107.stdout:Total 954 kB/s | 163 kB 00:00 2021-02-12T17:55:54.829 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:55:54.842 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:55:54.842 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:55:54.889 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:55:54.891 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:55:55.001 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:55:55.251 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-immutable-object-cache-2:17.0.0-713.g5a3430b9.e 1/1 2021-02-12T17:55:55.583 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.0.0-713.g5a3430b9.e 1/1 2021-02-12T17:55:55.836 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-immutable-object-cache-2:17.0.0-713.g5a3430b9.e 1/1 2021-02-12T17:55:55.837 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:55.837 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:55:55.837 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-immutable-object-cache-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:55:55.837 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:55.837 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:55:56.004 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr 2021-02-12T17:55:56.462 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:11 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:55:56.805 INFO:teuthology.orchestra.run.smithi107.stdout:Package ceph-mgr-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:55:56.918 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:55:56.919 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:55:56.919 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:55:56.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-dashboard 2021-02-12T17:55:57.446 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:12 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:55:57.916 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:55:57.917 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:57.917 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard noarch 2:17.0.0-713.g5a3430b9.el8 ceph-noarch 2.6 M 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards noarch 2:17.0.0-713.g5a3430b9.el8 ceph-noarch 24 k 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts noarch 2:17.0.0-713.g5a3430b9.el8 ceph-noarch 8.9 k 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt noarch 1.6.1-2.el8 CentOS-Base 43 k 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:55:57.918 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:55:57.919 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2021-02-12T17:55:57.919 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:55:57.919 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 2.9 M 2021-02-12T17:55:57.919 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 15 M 2021-02-12T17:55:57.919 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:55:58.071 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): ceph-prometheus-alerts-17.0.0-713.g5a343 59 kB/s | 8.9 kB 00:00 2021-02-12T17:55:58.092 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): ceph-grafana-dashboards-17.0.0-713.g5a34 140 kB/s | 24 kB 00:00 2021-02-12T17:55:58.170 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 436 kB/s | 43 kB 00:00 2021-02-12T17:55:58.205 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 298 kB/s | 34 kB 00:00 2021-02-12T17:55:58.270 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.9 MB/s | 196 kB 00:00 2021-02-12T17:55:58.294 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): ceph-mgr-dashboard-17.0.0-713.g5a3430b9. 7.1 MB/s | 2.6 MB 00:00 2021-02-12T17:55:58.297 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:55:58.297 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.8 MB/s | 2.9 MB 00:00 2021-02-12T17:55:58.299 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:55:58.308 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:55:58.309 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:55:58.472 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:55:58.474 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:55:58.963 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:55:59.157 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2021-02-12T17:55:59.435 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2021-02-12T17:55:59.677 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2021-02-12T17:55:59.798 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-prometheus-alerts-2:17.0.0-713.g5a3430b9.el8.no 4/6 2021-02-12T17:56:00.813 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-grafana-dashboards-2:17.0.0-713.g5a3430b9.el8.n 5/6 2021-02-12T17:56:01.084 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-dashboard-2:17.0.0-713.g5a3430b9.el8.noarch 6/6 2021-02-12T17:56:01.305 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.0.0-713.g5a3430b9.el8.noarch 6/6 2021-02-12T17:56:01.305 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-grafana-dashboards-2:17.0.0-713.g5a3430b9.el8.n 1/6 2021-02-12T17:56:01.305 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-dashboard-2:17.0.0-713.g5a3430b9.el8.noarch 2/6 2021-02-12T17:56:01.305 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-prometheus-alerts-2:17.0.0-713.g5a3430b9.el8.no 3/6 2021-02-12T17:56:01.305 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2021-02-12T17:56:01.305 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2021-02-12T17:56:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2021-02-12T17:56:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:01.594 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-grafana-dashboards-2:17.0.0-713.g5a3430b9.el8.noarch 2021-02-12T17:56:01.594 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-dashboard-2:17.0.0-713.g5a3430b9.el8.noarch 2021-02-12T17:56:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-prometheus-alerts-2:17.0.0-713.g5a3430b9.el8.noarch 2021-02-12T17:56:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jwt-1.6.1-2.el8.noarch 2021-02-12T17:56:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: python3-repoze-lru-0.7-6.el8.noarch 2021-02-12T17:56:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: python3-routes-2.4.1-12.el8.noarch 2021-02-12T17:56:01.595 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:01.595 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:01.654 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-diskprediction-local 2021-02-12T17:56:02.104 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:17 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:02.561 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:02.562 INFO:teuthology.orchestra.run.smithi107.stdout:===================================================================================================== 2021-02-12T17:56:02.562 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:56:02.562 INFO:teuthology.orchestra.run.smithi107.stdout:===================================================================================================== 2021-02-12T17:56:02.562 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:02.562 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local noarch 2:17.0.0-713.g5a3430b9.el8 ceph-noarch 8.6 M 2021-02-12T17:56:02.562 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:56:02.562 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran x86_64 8.3.1-5.1.el8 CentOS-Base 639 k 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath x86_64 8.3.1-5.1.el8 CentOS-Base 169 k 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout: openblas x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads x86_64 0.3.3-5.el8 CentOS-AppStream 4.3 M 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy x86_64 1:1.14.3-9.el8 CentOS-AppStream 3.7 M 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py x86_64 1:1.14.3-9.el8 CentOS-AppStream 225 k 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy x86_64 1.0.0-20.module_el8.3.0+562+e162826a CentOS-AppStream 15 M 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout:===================================================================================================== 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout:Install 8 Packages 2021-02-12T17:56:02.563 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:02.564 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 36 M 2021-02-12T17:56:02.564 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 194 M 2021-02-12T17:56:02.564 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:02.685 INFO:teuthology.orchestra.run.smithi107.stdout:(1/8): openblas-0.3.3-5.el8.x86_64.rpm 36 MB/s | 4.3 MB 00:00 2021-02-12T17:56:02.703 INFO:teuthology.orchestra.run.smithi107.stdout:(2/8): openblas-threads-0.3.3-5.el8.x86_64.rpm 31 MB/s | 4.3 MB 00:00 2021-02-12T17:56:02.710 INFO:teuthology.orchestra.run.smithi107.stdout:(3/8): python3-numpy-f2py-1.14.3-9.el8.x86_64.r 34 MB/s | 225 kB 00:00 2021-02-12T17:56:02.771 INFO:teuthology.orchestra.run.smithi107.stdout:(4/8): python3-numpy-1.14.3-9.el8.x86_64.rpm 43 MB/s | 3.7 MB 00:00 2021-02-12T17:56:02.787 INFO:teuthology.orchestra.run.smithi107.stdout:(5/8): libgfortran-8.3.1-5.1.el8.x86_64.rpm 40 MB/s | 639 kB 00:00 2021-02-12T17:56:02.793 INFO:teuthology.orchestra.run.smithi107.stdout:(6/8): libquadmath-8.3.1-5.1.el8.x86_64.rpm 28 MB/s | 169 kB 00:00 2021-02-12T17:56:03.015 INFO:teuthology.orchestra.run.smithi107.stdout:(7/8): python3-scipy-1.0.0-20.module_el8.3.0+56 48 MB/s | 15 MB 00:00 2021-02-12T17:56:03.101 INFO:teuthology.orchestra.run.smithi107.stdout:(8/8): ceph-mgr-diskprediction-local-17.0.0-713 16 MB/s | 8.6 MB 00:00 2021-02-12T17:56:03.104 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:03.104 INFO:teuthology.orchestra.run.smithi107.stdout:Total 67 MB/s | 36 MB 00:00 2021-02-12T17:56:03.112 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:03.132 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:03.132 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:03.420 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:03.422 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:04.677 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:05.324 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-02-12T17:56:05.455 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libquadmath-8.3.1-5.1.el8.x86_64 1/8 2021-02-12T17:56:05.607 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-02-12T17:56:06.114 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libgfortran-8.3.1-5.1.el8.x86_64 2/8 2021-02-12T17:56:06.274 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-0.3.3-5.el8.x86_64 3/8 2021-02-12T17:56:06.781 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-0.3.3-5.el8.x86_64 3/8 2021-02-12T17:56:06.931 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-02-12T17:56:07.677 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: openblas-threads-0.3.3-5.el8.x86_64 4/8 2021-02-12T17:56:08.013 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-1:1.14.3-9.el8.x86_64 5/8 2021-02-12T17:56:08.248 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-02-12T17:56:10.176 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 6/8 2021-02-12T17:56:11.734 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 7/8 2021-02-12T17:56:11.964 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-diskprediction-local-2:17.0.0-713.g5a3430b9 8/8 2021-02-12T17:56:12.205 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.0.0-713.g5a3430b9 8/8 2021-02-12T17:56:12.205 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.0.0-713.g5a3430b9 1/8 2021-02-12T17:56:12.205 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-0.3.3-5.el8.x86_64 2/8 2021-02-12T17:56:12.205 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : openblas-threads-0.3.3-5.el8.x86_64 3/8 2021-02-12T17:56:12.205 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-1:1.14.3-9.el8.x86_64 4/8 2021-02-12T17:56:12.205 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-numpy-f2py-1:1.14.3-9.el8.x86_64 5/8 2021-02-12T17:56:12.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x 6/8 2021-02-12T17:56:12.206 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libgfortran-8.3.1-5.1.el8.x86_64 7/8 2021-02-12T17:56:12.500 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libquadmath-8.3.1-5.1.el8.x86_64 8/8 2021-02-12T17:56:12.500 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:12.500 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-diskprediction-local-2:17.0.0-713.g5a3430b9.el8.noarch 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: libgfortran-8.3.1-5.1.el8.x86_64 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: libquadmath-8.3.1-5.1.el8.x86_64 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-0.3.3-5.el8.x86_64 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: openblas-threads-0.3.3-5.el8.x86_64 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-1:1.14.3-9.el8.x86_64 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: python3-numpy-f2py-1:1.14.3-9.el8.x86_64 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: python3-scipy-1.0.0-20.module_el8.3.0+562+e162826a.x86_64 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:12.501 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:12.590 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-rook 2021-02-12T17:56:13.039 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:28 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:13.516 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:13.518 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2021-02-12T17:56:13.518 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:56:13.518 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook noarch 2:17.0.0-713.g5a3430b9.el8 ceph-noarch 28 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch noarch 1.21-2.el8 CentOS-AppStream 27 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer noarch 1.10-11.el8 CentOS-AppStream 20 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib noarch 2.1.0-1.el8 CentOS-Base 155 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1 noarch 0.3.7-6.el8 CentOS-AppStream 126 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 CentOS-AppStream 110 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 CentOS-Base 43 k 2021-02-12T17:56:13.519 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa noarch 3.4.2-16.el8 epel 73 k 2021-02-12T17:56:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2021-02-12T17:56:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:13.520 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:13.520 INFO:teuthology.orchestra.run.smithi107.stdout:======================================================================================= 2021-02-12T17:56:13.520 INFO:teuthology.orchestra.run.smithi107.stdout:Install 13 Packages 2021-02-12T17:56:13.520 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:13.520 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 1.9 M 2021-02-12T17:56:13.520 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 24 M 2021-02-12T17:56:13.521 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:13.554 INFO:teuthology.orchestra.run.smithi107.stdout:(1/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 829 kB/s | 27 kB 00:00 2021-02-12T17:56:13.556 INFO:teuthology.orchestra.run.smithi107.stdout:(2/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 50 MB/s | 126 kB 00:00 2021-02-12T17:56:13.557 INFO:teuthology.orchestra.run.smithi107.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 568 kB/s | 20 kB 00:00 2021-02-12T17:56:13.559 INFO:teuthology.orchestra.run.smithi107.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 42 MB/s | 110 kB 00:00 2021-02-12T17:56:13.562 INFO:teuthology.orchestra.run.smithi107.stdout:(5/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 37 MB/s | 155 kB 00:00 2021-02-12T17:56:13.562 INFO:teuthology.orchestra.run.smithi107.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 19 MB/s | 43 kB 00:00 2021-02-12T17:56:13.622 INFO:teuthology.orchestra.run.smithi107.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 538 kB/s | 33 kB 00:00 2021-02-12T17:56:13.632 INFO:teuthology.orchestra.run.smithi107.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 231 kB/s | 16 kB 00:00 2021-02-12T17:56:13.645 INFO:teuthology.orchestra.run.smithi107.stdout:(9/13): ceph-mgr-rook-17.0.0-713.g5a3430b9.el8. 228 kB/s | 28 kB 00:00 2021-02-12T17:56:13.655 INFO:teuthology.orchestra.run.smithi107.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 2.4 MB/s | 81 kB 00:00 2021-02-12T17:56:13.672 INFO:teuthology.orchestra.run.smithi107.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 3.6 MB/s | 61 kB 00:00 2021-02-12T17:56:13.722 INFO:teuthology.orchestra.run.smithi107.stdout:(12/13): python3-rsa-3.4.2-16.el8.noarch.rpm 948 kB/s | 73 kB 00:00 2021-02-12T17:56:13.769 INFO:teuthology.orchestra.run.smithi107.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 8.5 MB/s | 1.2 MB 00:00 2021-02-12T17:56:13.770 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:13.771 INFO:teuthology.orchestra.run.smithi107.stdout:Total 7.7 MB/s | 1.9 MB 00:00 2021-02-12T17:56:13.776 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:13.790 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:13.790 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:13.859 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:13.862 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:14.410 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:14.675 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2021-02-12T17:56:14.912 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2021-02-12T17:56:15.142 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-rsa-3.4.2-16.el8.noarch 3/13 2021-02-12T17:56:15.387 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2021-02-12T17:56:15.603 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2021-02-12T17:56:15.827 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2021-02-12T17:56:17.055 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2021-02-12T17:56:17.330 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2021-02-12T17:56:18.431 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2021-02-12T17:56:18.753 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2021-02-12T17:56:18.945 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2021-02-12T17:56:19.120 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2021-02-12T17:56:19.312 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-rook-2:17.0.0-713.g5a3430b9.el8.noarch 13/13 2021-02-12T17:56:19.562 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-rook-2:17.0.0-713.g5a3430b9.el8.noarch 13/13 2021-02-12T17:56:19.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-rook-2:17.0.0-713.g5a3430b9.el8.noarch 1/13 2021-02-12T17:56:19.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2021-02-12T17:56:19.562 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2021-02-12T17:56:19.563 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-rsa-3.4.2-16.el8.noarch 12/13 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-rook-2:17.0.0-713.g5a3430b9.el8.noarch 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: python3-cachetools-3.1.1-4.el8.noarch 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: python3-certifi-2018.10.15-7.el8.noarch 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpatch-1.21-2.el8.noarch 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jsonpointer-1.10-11.el8.noarch 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2021-02-12T17:56:19.899 INFO:teuthology.orchestra.run.smithi107.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2021-02-12T17:56:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2021-02-12T17:56:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2021-02-12T17:56:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2021-02-12T17:56:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: python3-rsa-3.4.2-16.el8.noarch 2021-02-12T17:56:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2021-02-12T17:56:19.900 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:19.900 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:19.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-mgr-cephadm 2021-02-12T17:56:20.415 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:35 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:20.865 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:20.865 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:20.866 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:56:20.866 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:20.866 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:20.866 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm noarch 2:17.0.0-713.g5a3430b9.el8 ceph-noarch 79 k 2021-02-12T17:56:20.866 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:56:20.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2021-02-12T17:56:20.866 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel noarch 2.5.1-5.el8 CentOS-AppStream 4.8 M 2021-02-12T17:56:20.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2021-02-12T17:56:20.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2 noarch 2.10.1-2.el8_0 CentOS-AppStream 538 k 2021-02-12T17:56:20.867 INFO:teuthology.orchestra.run.smithi107.stdout: python3-remoto noarch 1.1.4-4.el8 epel 48 k 2021-02-12T17:56:20.867 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:20.867 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:20.867 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:20.867 INFO:teuthology.orchestra.run.smithi107.stdout:Install 6 Packages 2021-02-12T17:56:20.867 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:20.868 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 5.8 M 2021-02-12T17:56:20.868 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 24 M 2021-02-12T17:56:20.868 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:20.923 INFO:teuthology.orchestra.run.smithi107.stdout:(1/6): python3-jinja2-2.10.1-2.el8_0.noarch.rpm 9.6 MB/s | 538 kB 00:00 2021-02-12T17:56:20.966 INFO:teuthology.orchestra.run.smithi107.stdout:(2/6): python3-babel-2.5.1-5.el8.noarch.rpm 49 MB/s | 4.8 MB 00:00 2021-02-12T17:56:20.972 INFO:teuthology.orchestra.run.smithi107.stdout:(3/6): python3-apipkg-1.5-6.el8.noarch.rpm 431 kB/s | 21 kB 00:00 2021-02-12T17:56:21.005 INFO:teuthology.orchestra.run.smithi107.stdout:(4/6): python3-remoto-1.1.4-4.el8.noarch.rpm 1.4 MB/s | 48 kB 00:00 2021-02-12T17:56:21.012 INFO:teuthology.orchestra.run.smithi107.stdout:(5/6): ceph-mgr-cephadm-17.0.0-713.g5a3430b9.el 546 kB/s | 79 kB 00:00 2021-02-12T17:56:21.063 INFO:teuthology.orchestra.run.smithi107.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 3.6 MB/s | 355 kB 00:00 2021-02-12T17:56:21.065 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:21.065 INFO:teuthology.orchestra.run.smithi107.stdout:Total 29 MB/s | 5.8 MB 00:00 2021-02-12T17:56:21.068 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:21.078 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:21.078 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:21.154 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:21.156 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:21.454 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:21.697 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2021-02-12T17:56:21.991 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2021-02-12T17:56:23.501 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-remoto-1.1.4-4.el8.noarch 3/6 2021-02-12T17:56:23.903 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-babel-2.5.1-5.el8.noarch 4/6 2021-02-12T17:56:24.132 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : python3-jinja2-2.10.1-2.el8_0.noarch 5/6 2021-02-12T17:56:24.328 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-mgr-cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 6/6 2021-02-12T17:56:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 6/6 2021-02-12T17:56:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-mgr-cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 1/6 2021-02-12T17:56:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-babel-2.5.1-5.el8.noarch 2/6 2021-02-12T17:56:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-jinja2-2.10.1-2.el8_0.noarch 3/6 2021-02-12T17:56:24.595 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2021-02-12T17:56:24.596 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : python3-remoto-1.1.4-4.el8.noarch 6/6 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-mgr-cephadm-2:17.0.0-713.g5a3430b9.el8.noarch 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-apipkg-1.5-6.el8.noarch 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-babel-2.5.1-5.el8.noarch 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-execnet-1.7.1-1.el8.noarch 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-jinja2-2.10.1-2.el8_0.noarch 2021-02-12T17:56:24.889 INFO:teuthology.orchestra.run.smithi107.stdout: python3-remoto-1.1.4-4.el8.noarch 2021-02-12T17:56:24.890 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:24.890 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:24.955 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install ceph-fuse 2021-02-12T17:56:25.406 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:40 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:25.862 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:25.862 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repository Size 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 882 k 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout:Installing dependencies: 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout: fuse x86_64 2.9.7-12.el8 CentOS-Base 82 k 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-common x86_64 3.2.1-12.el8 CentOS-Base 21 k 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout:Install 3 Packages 2021-02-12T17:56:25.863 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:25.864 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 985 k 2021-02-12T17:56:25.864 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 3.2 M 2021-02-12T17:56:25.864 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:25.897 INFO:teuthology.orchestra.run.smithi107.stdout:(1/3): fuse-2.9.7-12.el8.x86_64.rpm 2.5 MB/s | 82 kB 00:00 2021-02-12T17:56:25.898 INFO:teuthology.orchestra.run.smithi107.stdout:(2/3): fuse-common-3.2.1-12.el8.x86_64.rpm 624 kB/s | 21 kB 00:00 2021-02-12T17:56:26.104 INFO:teuthology.orchestra.run.smithi107.stdout:(3/3): ceph-fuse-17.0.0-713.g5a3430b9.el8.x86_6 3.6 MB/s | 882 kB 00:00 2021-02-12T17:56:26.106 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:26.106 INFO:teuthology.orchestra.run.smithi107.stdout:Total 4.0 MB/s | 985 kB 00:00 2021-02-12T17:56:26.108 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:26.130 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:26.130 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:26.202 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:26.204 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:26.423 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:26.423 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-common-3.2.1-12.el8.x86_64 1/3 2021-02-12T17:56:26.423 INFO:teuthology.orchestra.run.smithi107.stdout:warning: /etc/fuse.conf created as /etc/fuse.conf.rpmnew 2021-02-12T17:56:26.423 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:26.903 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : fuse-2.9.7-12.el8.x86_64 2/3 2021-02-12T17:56:27.110 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : ceph-fuse-2:17.0.0-713.g5a3430b9.el8.x86_64 3/3 2021-02-12T17:56:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: ceph-fuse-2:17.0.0-713.g5a3430b9.el8.x86_64 3/3 2021-02-12T17:56:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : ceph-fuse-2:17.0.0-713.g5a3430b9.el8.x86_64 1/3 2021-02-12T17:56:27.464 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-2.9.7-12.el8.x86_64 2/3 2021-02-12T17:56:28.140 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : fuse-common-3.2.1-12.el8.x86_64 3/3 2021-02-12T17:56:28.140 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:28.140 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:28.140 INFO:teuthology.orchestra.run.smithi107.stdout: ceph-fuse-2:17.0.0-713.g5a3430b9.el8.x86_64 fuse-2.9.7-12.el8.x86_64 2021-02-12T17:56:28.140 INFO:teuthology.orchestra.run.smithi107.stdout: fuse-common-3.2.1-12.el8.x86_64 2021-02-12T17:56:28.140 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:28.140 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:28.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados-devel 2021-02-12T17:56:28.812 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:43 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:29.348 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 131 k 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:29.349 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2021-02-12T17:56:29.350 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:29.350 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 131 k 2021-02-12T17:56:29.350 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 437 k 2021-02-12T17:56:29.350 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:29.522 INFO:teuthology.orchestra.run.smithi107.stdout:librados-devel-17.0.0-713.g5a3430b9.el8.x86_64. 765 kB/s | 131 kB 00:00 2021-02-12T17:56:29.522 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:29.523 INFO:teuthology.orchestra.run.smithi107.stdout:Total 758 kB/s | 131 kB 00:00 2021-02-12T17:56:29.523 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:29.530 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:29.530 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:29.580 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:29.582 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:29.693 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:29.912 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : librados-devel-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:30.173 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: librados-devel-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:30.480 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : librados-devel-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:30.480 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:30.480 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:30.480 INFO:teuthology.orchestra.run.smithi107.stdout: librados-devel-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:56:30.480 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:30.481 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:30.683 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs2 2021-02-12T17:56:31.142 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:46 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:31.464 INFO:teuthology.orchestra.run.smithi107.stdout:Package libcephfs2-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:56:31.578 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:31.579 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:56:31.579 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:31.641 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install libcephfs-devel 2021-02-12T17:56:32.094 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:47 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:32.563 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:32.563 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2021-02-12T17:56:32.563 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:32.563 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 24 k 2021-02-12T17:56:32.563 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:32.564 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:32.564 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:32.564 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2021-02-12T17:56:32.564 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:32.564 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 24 k 2021-02-12T17:56:32.564 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 89 k 2021-02-12T17:56:32.564 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:32.689 INFO:teuthology.orchestra.run.smithi107.stdout:libcephfs-devel-17.0.0-713.g5a3430b9.el8.x86_64 193 kB/s | 24 kB 00:00 2021-02-12T17:56:32.689 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:32.689 INFO:teuthology.orchestra.run.smithi107.stdout:Total 191 kB/s | 24 kB 00:00 2021-02-12T17:56:32.690 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:32.694 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:32.694 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:32.696 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:32.699 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:32.752 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:32.941 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : libcephfs-devel-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:33.187 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: libcephfs-devel-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:33.587 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : libcephfs-devel-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:33.588 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: libcephfs-devel-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:56:33.588 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:33.588 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:33.652 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librados2 2021-02-12T17:56:34.104 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:49 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:34.455 INFO:teuthology.orchestra.run.smithi107.stdout:Package librados2-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:56:34.570 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:34.571 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:56:34.571 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:34.649 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install librbd1 2021-02-12T17:56:35.102 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:50 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:35.463 INFO:teuthology.orchestra.run.smithi107.stdout:Package librbd1-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:56:35.577 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:35.578 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:56:35.578 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:35.650 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rados 2021-02-12T17:56:36.102 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:51 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:36.429 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rados-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:56:36.547 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:36.548 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:56:36.548 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:36.608 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rgw 2021-02-12T17:56:37.058 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:52 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:37.413 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rgw-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:56:37.529 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:37.530 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:56:37.530 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:37.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-cephfs 2021-02-12T17:56:38.043 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:53 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:38.412 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-cephfs-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:56:38.525 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:38.526 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:56:38.526 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:38.588 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install python3-rbd 2021-02-12T17:56:39.044 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:54 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:39.714 INFO:teuthology.orchestra.run.smithi107.stdout:Package python3-rbd-2:17.0.0-713.g5a3430b9.el8.x86_64 is already installed. 2021-02-12T17:56:39.831 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:39.832 INFO:teuthology.orchestra.run.smithi107.stdout:Nothing to do. 2021-02-12T17:56:39.832 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:40.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-fuse 2021-02-12T17:56:40.604 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:55 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:41.101 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 89 k 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2021-02-12T17:56:41.102 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:41.103 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 89 k 2021-02-12T17:56:41.103 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 298 k 2021-02-12T17:56:41.103 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:41.249 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-fuse-17.0.0-713.g5a3430b9.el8.x86_64.rpm 611 kB/s | 89 kB 00:00 2021-02-12T17:56:41.249 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:41.249 INFO:teuthology.orchestra.run.smithi107.stdout:Total 606 kB/s | 89 kB 00:00 2021-02-12T17:56:41.249 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:41.261 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:41.262 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:41.312 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:41.314 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:41.453 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:41.673 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-fuse-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:41.903 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-fuse-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:42.193 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-fuse-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:42.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:42.194 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:42.194 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-fuse-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:56:42.194 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:42.194 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:42.390 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-mirror 2021-02-12T17:56:42.848 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:01:57 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:43.318 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:43.318 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: Package Arch Version Repo Size 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 3.7 M 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 3.7 M 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 18 M 2021-02-12T17:56:43.319 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:43.655 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-mirror-17.0.0-713.g5a3430b9.el8.x86_64.rpm 11 MB/s | 3.7 MB 00:00 2021-02-12T17:56:43.655 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:43.655 INFO:teuthology.orchestra.run.smithi107.stdout:Total 11 MB/s | 3.7 MB 00:00 2021-02-12T17:56:43.655 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:43.668 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:43.668 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:43.731 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:43.733 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:44.239 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:44.457 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-mirror-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:44.824 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-mirror-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:45.093 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-mirror-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:45.093 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:45.093 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:45.093 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-mirror-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:56:45.093 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:45.093 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:45.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo yum -y install rbd-nbd 2021-02-12T17:56:45.723 INFO:teuthology.orchestra.run.smithi107.stdout:Last metadata expiration check: 0:02:00 ago on Fri 12 Feb 2021 05:54:45 PM UTC. 2021-02-12T17:56:46.549 INFO:teuthology.orchestra.run.smithi107.stdout:Dependencies resolved. 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout: Package Architecture Version Repository Size 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:Installing: 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd x86_64 2:17.0.0-713.g5a3430b9.el8 ceph 174 k 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction Summary 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:================================================================================ 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:Install 1 Package 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:Total download size: 174 k 2021-02-12T17:56:46.550 INFO:teuthology.orchestra.run.smithi107.stdout:Installed size: 649 k 2021-02-12T17:56:46.551 INFO:teuthology.orchestra.run.smithi107.stdout:Downloading Packages: 2021-02-12T17:56:46.717 INFO:teuthology.orchestra.run.smithi107.stdout:rbd-nbd-17.0.0-713.g5a3430b9.el8.x86_64.rpm 1.0 MB/s | 174 kB 00:00 2021-02-12T17:56:46.718 INFO:teuthology.orchestra.run.smithi107.stdout:-------------------------------------------------------------------------------- 2021-02-12T17:56:46.718 INFO:teuthology.orchestra.run.smithi107.stdout:Total 1.0 MB/s | 174 kB 00:00 2021-02-12T17:56:46.719 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction check 2021-02-12T17:56:46.733 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction check succeeded. 2021-02-12T17:56:46.733 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction test 2021-02-12T17:56:46.784 INFO:teuthology.orchestra.run.smithi107.stdout:Transaction test succeeded. 2021-02-12T17:56:46.787 INFO:teuthology.orchestra.run.smithi107.stdout:Running transaction 2021-02-12T17:56:46.947 INFO:teuthology.orchestra.run.smithi107.stdout: Preparing : 1/1 2021-02-12T17:56:47.180 INFO:teuthology.orchestra.run.smithi107.stdout: Installing : rbd-nbd-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:47.397 INFO:teuthology.orchestra.run.smithi107.stdout: Running scriptlet: rbd-nbd-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: Verifying : rbd-nbd-2:17.0.0-713.g5a3430b9.el8.x86_64 1/1 2021-02-12T17:56:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:47.728 INFO:teuthology.orchestra.run.smithi107.stdout:Installed: 2021-02-12T17:56:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: rbd-nbd-2:17.0.0-713.g5a3430b9.el8.x86_64 2021-02-12T17:56:47.728 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:47.728 INFO:teuthology.orchestra.run.smithi107.stdout:Complete! 2021-02-12T17:56:47.886 DEBUG:teuthology.parallel:result is None 2021-02-12T17:56:47.887 DEBUG:teuthology.orchestra.run.smithi107:> sudo lsb_release -is 2021-02-12T17:56:47.920 INFO:teuthology.orchestra.run.smithi107.stdout:CentOS 2021-02-12T17:56:47.921 DEBUG:teuthology.misc:System to be installed: CentOS 2021-02-12T17:56:47.922 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=5a3430b93fcfcd31d38534758b8997c894688c4b 2021-02-12T17:56:48.093 DEBUG:teuthology.orchestra.run.smithi107:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2021-02-12T17:56:48.112 INFO:teuthology.orchestra.run.smithi107.stdout:17.0.0-713.g5a3430b9.el8 2021-02-12T17:56:48.113 INFO:teuthology.packaging:The installed version of ceph is 17.0.0-713.g5a3430b9.el8 2021-02-12T17:56:48.113 INFO:teuthology.task.install:The correct ceph version 17.0.0-713.g5a3430b9 is installed. 2021-02-12T17:56:48.113 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2021-02-12T17:56:48.113 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:48.113 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2021-02-12T17:56:48.176 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2021-02-12T17:56:48.177 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:48.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/daemon-helper 2021-02-12T17:56:48.240 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/daemon-helper 2021-02-12T17:56:48.304 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2021-02-12T17:56:48.304 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:48.304 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/usr/bin/adjust-ulimits 2021-02-12T17:56:48.368 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2021-02-12T17:56:48.433 INFO:teuthology.run_tasks:Running task ceph... 2021-02-12T17:56:48.490 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2021-02-12T17:56:48.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 777 /var/log/ceph 2021-02-12T17:56:48.515 INFO:tasks.ceph:Disabling ceph logrotate... 2021-02-12T17:56:48.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/logrotate.d/ceph 2021-02-12T17:56:48.579 INFO:tasks.ceph:Creating extra log directories... 2021-02-12T17:56:48.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2021-02-12T17:56:48.651 INFO:tasks.ceph:Creating ceph cluster ceph... 2021-02-12T17:56:48.652 INFO:tasks.ceph:config {'conf': {'osd': {'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': False, 'mon_bind_addrvec': True} 2021-02-12T17:56:48.652 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860', 'branch': 'wip-dmclock-fix-high-client-profile', 'description': 'rados/singleton/{all/osd-backfill mon_election/connectivity msgr-failures/many msgr/async-v1only objectstore/bluestore-low-osd-mem-target rados supported-random-distro$/{centos_8}}', 'email': None, 'first_in_suite': False, 'job_id': '5877860', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'machine_type': 'smithi', 'name': 'sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi', 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 3, 'size': 10}}], 'os_type': 'centos', 'os_version': '8.2', 'overrides': {'admin_socket': {'branch': 'wip-dmclock-fix-high-client-profile'}, 'ceph': {'conf': {'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}}, 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}}}, 'install': {'ceph': {'sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b'}}, 'selinux': {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'wip-dmclock-fix-high-client-profile', 'sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b'}}, 'owner': 'scheduled_sseshasa@teuthology', 'priority': 70, 'repo': 'git://git.ceph.com/ceph-ci.git', 'roles': [['mon.a', 'mon.b', 'mon.c', 'mgr.x', 'osd.0', 'osd.1', 'osd.2']], 'sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b', 'sleep_before_teardown': 0, 'suite': 'rados', 'suite_branch': 'wip-dmclock-fix-high-client-profile', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa', 'suite_relpath': 'qa', 'suite_repo': 'git://git.ceph.com/ceph-ci.git', 'suite_sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b', 'targets': {'smithi107.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCU7A7ySnMnqXkw6mW2Ut7f3wWgG3dEhpElLMt5wQt49bDKJZh1Geh7/fnEHP6nS1p/cc6LPv9crBJShjp2APgkTMzAr2xPb30BJsOV6MpevHZRwE3tgfGXAp6Hyqzz5+iMbbnvb1oZRp/mYhO+aNuFH++RcohkB0LDBVTC1kz+yfIzFJk7P//V53ykuf77O/8LNBfh0ZaJ8reur1K1ppwfU6fOG9wbcL3GZRraETHDm1lFabdMLpqiU5roPqKff4IYdZnCFyvgRTlRsH9AjgEWsvSWs6IHsoswihwQsSXZy416u6twSN56MFc2yArCaGKH6B1f7okbvXbmx2Kzhh5U6Fssu/ajVjl663/L88eXDr5pEIbjXCkAGAV8/PID+/xe7qqH4fKAd2XojJ8mgGKdk2DOR8DvQDlekCzlIeqD0JWKS3xeylAtXoA/ypFZIuxYspEJ/WWAwjSyWm/6LyIn+6XjYgD8+pzkpNa2w6bjbBos0r4dOtctSfhg4UyerPk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': None}, {'ceph': {'conf': {'osd': {'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'global': {'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'mgr': {'debug mgr': 20, 'debug monc': 10, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon scrub interval': 300}}, 'log-ignorelist': ['but it is still running', 'overall HEALTH_', '\\(OSDMAP_FLAGS\\)', '\\(OSD_', '\\(PG_', '\\(OBJECT_', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'pre-mgr-commands': ['sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force'], 'fs': 'xfs', 'log-whitelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)'], 'mon_bind_msgr2': False, 'sha1': '5a3430b93fcfcd31d38534758b8997c894688c4b', 'cluster': 'ceph'}}, {'osd_backfill': None}], 'teuthology_branch': 'master', 'teuthology_sha1': '87d40f79d7da6caf6da06c190938773bf745140c', 'timestamp': '2021-02-12_17:37:51', 'tube': 'smithi', 'user': 'sseshasa', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.29888'} 2021-02-12T17:56:48.652 DEBUG:teuthology.orchestra.run.smithi107:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2021-02-12T17:56:48.698 DEBUG:teuthology.orchestra.run.smithi107:> sudo install -d -m0777 -- /var/run/ceph 2021-02-12T17:56:48.762 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:48.762 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/scratch_devs of=/dev/stdout 2021-02-12T17:56:48.815 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2021-02-12T17:56:48.815 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2021-02-12T17:56:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2021-02-12T17:56:48.869 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-12T17:56:48.869 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 114058 Links: 1 2021-02-12T17:56:48.869 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-12T17:56:48.870 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2021-02-12T17:56:48.870 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2021-02-12 17:56:44.631820519 +0000 2021-02-12T17:56:48.870 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2021-02-12 17:54:18.498618537 +0000 2021-02-12T17:56:48.870 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2021-02-12 17:54:18.498618537 +0000 2021-02-12T17:56:48.870 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2021-02-12T17:56:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2021-02-12T17:56:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2021-02-12T17:56:48.879 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000231679 s, 2.2 MB/s 2021-02-12T17:56:48.882 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 114188 Links: 1 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2021-02-12 17:56:44.632820503 +0000 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2021-02-12 17:54:18.844612943 +0000 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2021-02-12 17:54:18.844612943 +0000 2021-02-12T17:56:48.937 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2021-02-12T17:56:48.946 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2021-02-12T17:56:48.947 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2021-02-12T17:56:48.947 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000236279 s, 2.2 MB/s 2021-02-12T17:56:48.949 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2021-02-12T17:56:49.003 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2021-02-12T17:56:49.003 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-12T17:56:49.004 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 110350 Links: 1 2021-02-12T17:56:49.004 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-12T17:56:49.004 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2021-02-12T17:56:49.004 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2021-02-12 17:56:44.632820503 +0000 2021-02-12T17:56:49.004 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2021-02-12 17:54:19.165607753 +0000 2021-02-12T17:56:49.004 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2021-02-12 17:54:19.165607753 +0000 2021-02-12T17:56:49.004 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2021-02-12T17:56:49.013 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2021-02-12T17:56:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2021-02-12T17:56:49.014 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000209406 s, 2.4 MB/s 2021-02-12T17:56:49.016 DEBUG:teuthology.orchestra.run.smithi107:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2021-02-12T17:56:49.070 INFO:teuthology.orchestra.run.smithi107.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2021-02-12T17:56:49.070 INFO:teuthology.orchestra.run.smithi107.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2021-02-12T17:56:49.071 INFO:teuthology.orchestra.run.smithi107.stdout:Device: 6h/6d Inode: 112403 Links: 1 2021-02-12T17:56:49.071 INFO:teuthology.orchestra.run.smithi107.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2021-02-12T17:56:49.071 INFO:teuthology.orchestra.run.smithi107.stdout:Context: system_u:object_r:device_t:s0 2021-02-12T17:56:49.071 INFO:teuthology.orchestra.run.smithi107.stdout:Access: 2021-02-12 17:56:44.632820503 +0000 2021-02-12T17:56:49.071 INFO:teuthology.orchestra.run.smithi107.stdout:Modify: 2021-02-12 17:54:19.525601932 +0000 2021-02-12T17:56:49.071 INFO:teuthology.orchestra.run.smithi107.stdout:Change: 2021-02-12 17:54:19.525601932 +0000 2021-02-12T17:56:49.071 INFO:teuthology.orchestra.run.smithi107.stdout: Birth: - 2021-02-12T17:56:49.080 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records in 2021-02-12T17:56:49.081 INFO:teuthology.orchestra.run.smithi107.stderr:1+0 records out 2021-02-12T17:56:49.081 INFO:teuthology.orchestra.run.smithi107.stderr:512 bytes copied, 0.000201379 s, 2.5 MB/s 2021-02-12T17:56:49.084 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-02-12T17:56:49.084 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-12T17:56:49.084 INFO:tasks.ceph:Generating config... 2021-02-12T17:56:49.086 INFO:tasks.ceph:[osd] osd min pg log entries = 5 2021-02-12T17:56:49.086 INFO:tasks.ceph:[osd] bdev async discard = True 2021-02-12T17:56:49.086 INFO:tasks.ceph:[osd] bdev enable discard = True 2021-02-12T17:56:49.086 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2021-02-12T17:56:49.087 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2021-02-12T17:56:49.087 INFO:tasks.ceph:[osd] debug bluefs = 20 2021-02-12T17:56:49.087 INFO:tasks.ceph:[osd] debug bluestore = 20 2021-02-12T17:56:49.087 INFO:tasks.ceph:[osd] debug filestore = 20 2021-02-12T17:56:49.087 INFO:tasks.ceph:[osd] debug journal = 20 2021-02-12T17:56:49.087 INFO:tasks.ceph:[osd] debug ms = 20 2021-02-12T17:56:49.088 INFO:tasks.ceph:[osd] debug osd = 25 2021-02-12T17:56:49.088 INFO:tasks.ceph:[osd] debug rocksdb = 10 2021-02-12T17:56:49.088 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2021-02-12T17:56:49.088 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2021-02-12T17:56:49.088 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2021-02-12T17:56:49.088 INFO:tasks.ceph:[osd] osd debug verify cached snaps = True 2021-02-12T17:56:49.089 INFO:tasks.ceph:[osd] osd debug verify missing on start = True 2021-02-12T17:56:49.089 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2021-02-12T17:56:49.089 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2021-02-12T17:56:49.089 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2021-02-12T17:56:49.089 INFO:tasks.ceph:[osd] osd op queue = debug_random 2021-02-12T17:56:49.089 INFO:tasks.ceph:[osd] osd op queue cut off = debug_random 2021-02-12T17:56:49.089 INFO:tasks.ceph:[global] mon client directed command retry = 5 2021-02-12T17:56:49.090 INFO:tasks.ceph:[global] mon client hunt interval max multiple = 2 2021-02-12T17:56:49.090 INFO:tasks.ceph:[global] mon election default strategy = 3 2021-02-12T17:56:49.090 INFO:tasks.ceph:[global] mon mgr beacon grace = 90 2021-02-12T17:56:49.090 INFO:tasks.ceph:[global] ms bind msgr2 = False 2021-02-12T17:56:49.090 INFO:tasks.ceph:[global] ms inject socket failures = 1000 2021-02-12T17:56:49.090 INFO:tasks.ceph:[global] ms type = async 2021-02-12T17:56:49.090 INFO:tasks.ceph:[mgr] debug mgr = 20 2021-02-12T17:56:49.090 INFO:tasks.ceph:[mgr] debug monc = 10 2021-02-12T17:56:49.091 INFO:tasks.ceph:[mgr] debug ms = 1 2021-02-12T17:56:49.091 INFO:tasks.ceph:[mon] debug mon = 20 2021-02-12T17:56:49.091 INFO:tasks.ceph:[mon] debug ms = 1 2021-02-12T17:56:49.091 INFO:tasks.ceph:[mon] debug paxos = 20 2021-02-12T17:56:49.091 INFO:tasks.ceph:[mon] mon scrub interval = 300 2021-02-12T17:56:49.091 INFO:tasks.ceph:Setting up mon.a... 2021-02-12T17:56:49.091 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2021-02-12T17:56:49.196 INFO:teuthology.orchestra.run.smithi107.stdout:creating /etc/ceph/ceph.keyring 2021-02-12T17:56:49.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2021-02-12T17:56:49.237 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-12T17:56:49.302 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '[v1:172.21.15.107:6789]'), ('mon.b', '[v1:172.21.15.107:6790]'), ('mon.c', '[v1:172.21.15.107:6791]')] 2021-02-12T17:56:49.303 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '[v1:172.21.15.107:6789],[v1:172.21.15.107:6790],[v1:172.21.15.107:6791]', 'mon client directed command retry': 5, 'mon client hunt interval max multiple': 2, 'mon election default strategy': 3, 'mon mgr beacon grace': 90, 'ms bind msgr2': False, 'ms inject socket failures': 1000, 'ms type': 'async'}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'osd min pg log entries': 5, 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': 20, 'debug bluestore': 20, 'debug filestore': 20, 'debug journal': 20, 'debug ms': 20, 'debug osd': 25, 'debug rocksdb': 10, 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd debug verify cached snaps': True, 'osd debug verify missing on start': True, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore', 'osd op queue': 'debug_random', 'osd op queue cut off': 'debug_random'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false', 'debug monc': 10}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'mon scrub interval': 300}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok'}, 'mon.a': {}, 'mon.b': {}, 'mon.c': {}} 2021-02-12T17:56:49.305 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:49.305 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2021-02-12T17:56:49.356 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --addv a '[v1:172.21.15.107:6789]' --addv b '[v1:172.21.15.107:6790]' --addv c '[v1:172.21.15.107:6791]' --print /home/ubuntu/cephtest/ceph.monmap 2021-02-12T17:56:49.437 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2021-02-12T17:56:49.437 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: generated fsid 9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c 2021-02-12T17:56:49.437 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 0 2021-02-12T17:56:49.437 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c 2021-02-12T17:56:49.437 INFO:teuthology.orchestra.run.smithi107.stdout:last_changed 2021-02-12T17:56:49.437374+0000 2021-02-12T17:56:49.437 INFO:teuthology.orchestra.run.smithi107.stdout:created 2021-02-12T17:56:49.437374+0000 2021-02-12T17:56:49.437 INFO:teuthology.orchestra.run.smithi107.stdout:min_mon_release 0 (unknown) 2021-02-12T17:56:49.438 INFO:teuthology.orchestra.run.smithi107.stdout:election_strategy: 3 2021-02-12T17:56:49.438 INFO:teuthology.orchestra.run.smithi107.stdout:0: v1:172.21.15.107:6789/0 mon.a 2021-02-12T17:56:49.438 INFO:teuthology.orchestra.run.smithi107.stdout:1: v1:172.21.15.107:6790/0 mon.b 2021-02-12T17:56:49.438 INFO:teuthology.orchestra.run.smithi107.stdout:2: v1:172.21.15.107:6791/0 mon.c 2021-02-12T17:56:49.438 INFO:teuthology.orchestra.run.smithi107.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2021-02-12T17:56:49.439 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2021-02-12T17:56:49.494 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c... 2021-02-12T17:56:49.495 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2021-02-12T17:56:49.577 INFO:teuthology.orchestra.run.smithi107.stdout:[global] 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: chdir = "" 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: auth supported = cephx 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: filestore xattr use omap = true 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: mon clock drift allowed = 1.000 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: osd crush chooseleaf type = 0 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: auth debug = true 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.578 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on old message = true 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: ms die on bug = true 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon max pg per osd = 10000 # >= luminous 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon pg warn max object skew = 0 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: # disable pg_autoscaler by default for new pools 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: osd_pool_default_pg_autoscale_mode = off 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default size = 2 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow primary affinity = true 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd allow pg remap = true 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on legacy crush tunables = false 2021-02-12T17:56:49.579 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on crush straw calc version zero = false 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on no sortbitwise = false 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on osd down out interval zero = false 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: mon warn on too few osds = false 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: mon_warn_on_pool_no_redundancy = false 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: mon_allow_pool_size_one = true 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: osd default data pool replay window = 5 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: mon allow pool delete = true 2021-02-12T17:56:49.580 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: mon cluster log file level = debug 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: debug asserts on shutdown = true 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: mon health detail to clog = false 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: mon host = "[v1:172.21.15.107:6789],[v1:172.21.15.107:6790],[v1:172.21.15.107:6791]" 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: mon client directed command retry = 5 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: mon client hunt interval max multiple = 2 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: mon election default strategy = 3 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr beacon grace = 90 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: ms bind msgr2 = False 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: ms inject socket failures = 1000 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: ms type = async 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: fsid = 9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.581 INFO:teuthology.orchestra.run.smithi107.stdout:[osd] 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd journal size = 100 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub load threshold = 5.0 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd scrub max interval = 600 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd recover clone overlap = true 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd recovery max chunk = 1048576 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug shutdown = true 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug op order = true 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify stray on activate = true 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.582 INFO:teuthology.orchestra.run.smithi107.stdout: osd open classes on start = true 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug pg log writeout = true 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: osd deep scrub update digest min age = 30 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: osd map max advance = 10 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: journal zero on create = true 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: filestore ondisk finisher threads = 3 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: filestore apply finisher threads = 3 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: bdev debug aio = true 2021-02-12T17:56:49.583 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug misdirected ops = true 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: osd min pg log entries = 5 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: bdev async discard = True 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: bdev enable discard = True 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore block size = 96636764160 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: bluestore fsck on mount = True 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluefs = 20 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug bluestore = 20 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug filestore = 20 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug journal = 20 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 20 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug osd = 25 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: debug rocksdb = 10 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd backfillfull_ratio = 0.85 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd full ratio = 0.9 2021-02-12T17:56:49.584 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd nearfull ratio = 0.8 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify cached snaps = True 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: osd debug verify missing on start = True 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: osd failsafe full ratio = 0.95 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: osd memory target = 1610612736 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: osd objectstore = bluestore 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue = debug_random 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: osd op queue cut off = debug_random 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr] 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: debug mgr = 20 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2021-02-12T17:56:49.585 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: mgr/telemetry/nag = false 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug monc = 10 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout:[mon] 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug ms = 1 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug mon = 20 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug paxos = 20 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: debug auth = 20 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: mon data avail warn = 5 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: mon mgr mkfs grace = 240 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min pgs per osd = 4 2021-02-12T17:56:49.586 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd reporter subtree level = osd 2021-02-12T17:56:49.587 INFO:teuthology.orchestra.run.smithi107.stdout: mon osd prime pg temp = true 2021-02-12T17:56:49.587 INFO:teuthology.orchestra.run.smithi107.stdout: mon reweight min bytes per osd = 10 2021-02-12T17:56:49.587 INFO:teuthology.orchestra.run.smithi107.stdout: mon scrub interval = 300 2021-02-12T17:56:49.587 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:56:49.587 INFO:teuthology.orchestra.run.smithi107.stdout:[client] 2021-02-12T17:56:49.587 INFO:teuthology.orchestra.run.smithi107.stdout: rgw cache enabled = true 2021-02-12T17:56:49.587 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable ops log = true 2021-02-12T17:56:49.587 INFO:teuthology.orchestra.run.smithi107.stdout: rgw enable usage log = true 2021-02-12T17:56:49.588 INFO:teuthology.orchestra.run.smithi107.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2021-02-12T17:56:49.588 INFO:teuthology.orchestra.run.smithi107.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2021-02-12T17:56:49.588 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.a] 2021-02-12T17:56:49.588 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.b] 2021-02-12T17:56:49.588 INFO:teuthology.orchestra.run.smithi107.stdout:[mon.c] 2021-02-12T17:56:49.589 INFO:tasks.ceph:Creating admin key on mon.a... 2021-02-12T17:56:49.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2021-02-12T17:56:49.638 INFO:tasks.ceph:Copying monmap to all nodes... 2021-02-12T17:56:49.638 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:49.638 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/etc/ceph/ceph.keyring of=/dev/stdout 2021-02-12T17:56:49.652 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:49.652 DEBUG:teuthology.orchestra.run.smithi107:> dd if=/home/ubuntu/cephtest/ceph.monmap of=/dev/stdout 2021-02-12T17:56:49.707 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:56:49.707 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:49.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd of=/etc/ceph/ceph.keyring 2021-02-12T17:56:49.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0644 /etc/ceph/ceph.keyring 2021-02-12T17:56:49.782 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:56:49.782 DEBUG:teuthology.orchestra.run.smithi107:> dd of=/home/ubuntu/cephtest/ceph.monmap 2021-02-12T17:56:49.835 INFO:tasks.ceph:Setting up mon nodes... 2021-02-12T17:56:49.835 INFO:tasks.ceph:Setting up mgr nodes... 2021-02-12T17:56:49.836 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2021-02-12T17:56:49.923 INFO:teuthology.orchestra.run.smithi107.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2021-02-12T17:56:49.925 INFO:tasks.ceph:Setting up mds nodes... 2021-02-12T17:56:49.925 INFO:tasks.ceph_client:Setting up client nodes... 2021-02-12T17:56:49.925 INFO:tasks.ceph:Running mkfs on osd nodes... 2021-02-12T17:56:49.925 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi107.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'}} 2021-02-12T17:56:49.925 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2021-02-12T17:56:49.949 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-02-12T17:56:49.949 INFO:tasks.ceph:role: osd.0 2021-02-12T17:56:49.949 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:56:49.950 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2021-02-12T17:56:50.021 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2021-02-12T17:56:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-12T17:56:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-02-12T17:56:50.021 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2021-02-12T17:56:50.022 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-12T17:56:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2021-02-12T17:56:50.022 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-02-12T17:56:50.022 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-12T17:56:50.022 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-12T17:56:50.022 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-12T17:56:50.380 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2021-02-12T17:56:50.382 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2021-02-12T17:56:50.382 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2021-02-12T17:56:50.515 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2021-02-12T17:56:50.544 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2021-02-12T17:56:50.609 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-02-12T17:56:50.609 INFO:tasks.ceph:role: osd.1 2021-02-12T17:56:50.609 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:56:50.609 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2021-02-12T17:56:50.681 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2021-02-12T17:56:50.681 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-12T17:56:50.681 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-02-12T17:56:50.681 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2021-02-12T17:56:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-12T17:56:50.682 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2021-02-12T17:56:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-02-12T17:56:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-12T17:56:50.682 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-12T17:56:50.682 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-12T17:56:51.046 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2021-02-12T17:56:51.578 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2021-02-12T17:56:51.579 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2021-02-12T17:56:51.665 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2021-02-12T17:56:51.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2021-02-12T17:56:51.761 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3'} 2021-02-12T17:56:51.761 INFO:tasks.ceph:role: osd.2 2021-02-12T17:56:51.761 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T17:56:51.761 DEBUG:teuthology.orchestra.run.smithi107:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2021-02-12T17:56:51.834 INFO:teuthology.orchestra.run.smithi107.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2021-02-12T17:56:51.834 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 attr=2, projid32bit=1 2021-02-12T17:56:51.834 INFO:teuthology.orchestra.run.smithi107.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2021-02-12T17:56:51.834 INFO:teuthology.orchestra.run.smithi107.stdout: = reflink=1 2021-02-12T17:56:51.834 INFO:teuthology.orchestra.run.smithi107.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2021-02-12T17:56:51.834 INFO:teuthology.orchestra.run.smithi107.stdout: = sunit=0 swidth=0 blks 2021-02-12T17:56:51.834 INFO:teuthology.orchestra.run.smithi107.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2021-02-12T17:56:51.834 INFO:teuthology.orchestra.run.smithi107.stdout:log =internal log bsize=4096 blocks=11443, version=2 2021-02-12T17:56:51.835 INFO:teuthology.orchestra.run.smithi107.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2021-02-12T17:56:51.835 INFO:teuthology.orchestra.run.smithi107.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2021-02-12T17:56:52.205 INFO:teuthology.orchestra.run.smithi107.stdout:Discarding blocks...Done. 2021-02-12T17:56:52.207 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi107.front.sepia.ceph.com -o noatime 2021-02-12T17:56:52.207 DEBUG:teuthology.orchestra.run.smithi107:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2021-02-12T17:56:52.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2021-02-12T17:56:52.342 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-12T17:56:52.437 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:52.435+0000 7fd68388b080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2021-02-12T17:56:52.437 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:52.436+0000 7fd68388b080 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2021-02-12T17:56:52.437 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:52.436+0000 7fd68388b080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-12T17:56:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:52.436+0000 7fd68388b080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-12T17:56:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:52.436+0000 7fd68388b080 -1 bluestore(/var/lib/ceph/osd/ceph-0/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-0/block: (2) No such file or directory 2021-02-12T17:56:52.438 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:52.437+0000 7fd68388b080 -1 bluestore(/var/lib/ceph/osd/ceph-0) _read_fsid unparsable uuid 2021-02-12T17:56:55.279 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2021-02-12T17:56:55.309 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-12T17:56:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:55.394+0000 7f8c740e6080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2021-02-12T17:56:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:55.394+0000 7f8c740e6080 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2021-02-12T17:56:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:55.394+0000 7f8c740e6080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-12T17:56:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:55.394+0000 7f8c740e6080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-12T17:56:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:55.394+0000 7f8c740e6080 -1 bluestore(/var/lib/ceph/osd/ceph-1/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-1/block: (2) No such file or directory 2021-02-12T17:56:55.395 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:55.394+0000 7f8c740e6080 -1 bluestore(/var/lib/ceph/osd/ceph-1) _read_fsid unparsable uuid 2021-02-12T17:56:58.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2021-02-12T17:56:58.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2021-02-12T17:56:58.351 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:58.350+0000 7f81bb70c080 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2021-02-12T17:56:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:58.350+0000 7f81bb70c080 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2021-02-12T17:56:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:58.350+0000 7f81bb70c080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-12T17:56:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:58.350+0000 7f81bb70c080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-12T17:56:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:58.350+0000 7f81bb70c080 -1 bluestore(/var/lib/ceph/osd/ceph-2/block) _read_bdev_label failed to open /var/lib/ceph/osd/ceph-2/block: (2) No such file or directory 2021-02-12T17:56:58.352 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:56:58.351+0000 7f81bb70c080 -1 bluestore(/var/lib/ceph/osd/ceph-2) _read_fsid unparsable uuid 2021-02-12T17:57:01.196 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2021-02-12T17:57:01.225 INFO:tasks.ceph:Reading keys from all nodes... 2021-02-12T17:57:01.226 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:57:01.226 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/mgr/ceph-x/keyring of=/dev/stdout 2021-02-12T17:57:01.293 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:57:01.293 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/keyring of=/dev/stdout 2021-02-12T17:57:01.360 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:57:01.360 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/keyring of=/dev/stdout 2021-02-12T17:57:01.426 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:57:01.426 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/keyring of=/dev/stdout 2021-02-12T17:57:01.492 INFO:tasks.ceph:Adding keys to all mons... 2021-02-12T17:57:01.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo tee -a /etc/ceph/ceph.keyring 2021-02-12T17:57:01.556 INFO:teuthology.orchestra.run.smithi107.stdout:[mgr.x] 2021-02-12T17:57:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBhwSZgMhQKNxAAcPcji7YXZD9blFHvU2EOtw== 2021-02-12T17:57:01.556 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.0] 2021-02-12T17:57:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBkwSZgVp8UGhAAGj6PmRBpDbgODRBSWcSNCQ== 2021-02-12T17:57:01.556 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.1] 2021-02-12T17:57:01.556 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBnwSZgb8WHFxAAHAVV7slOcGkc51p2V/Mryg== 2021-02-12T17:57:01.557 INFO:teuthology.orchestra.run.smithi107.stdout:[osd.2] 2021-02-12T17:57:01.557 INFO:teuthology.orchestra.run.smithi107.stdout: key = AQBqwSZgitnsFBAAJemEqRD3zI0WubOnEXeyGQ== 2021-02-12T17:57:01.557 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2021-02-12T17:57:01.644 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-12T17:57:01.689 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-12T17:57:01.735 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2021-02-12T17:57:01.781 INFO:tasks.ceph:Running mkfs on mon nodes... 2021-02-12T17:57:01.781 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2021-02-12T17:57:01.806 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-12T17:57:01.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2021-02-12T17:57:01.923 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2021-02-12T17:57:01.988 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-12T17:57:02.078 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2021-02-12T17:57:02.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2021-02-12T17:57:02.166 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2021-02-12T17:57:02.260 DEBUG:teuthology.orchestra.run.smithi107:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2021-02-12T17:57:02.284 DEBUG:teuthology.orchestra.run.smithi107:> rm -- /home/ubuntu/cephtest/ceph.monmap 2021-02-12T17:57:02.338 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2021-02-12T17:57:02.339 INFO:tasks.ceph.mon.a:Restarting daemon 2021-02-12T17:57:02.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2021-02-12T17:57:02.381 INFO:tasks.ceph.mon.a:Started 2021-02-12T17:57:02.381 INFO:tasks.ceph.mon.b:Restarting daemon 2021-02-12T17:57:02.381 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2021-02-12T17:57:02.383 INFO:tasks.ceph.mon.b:Started 2021-02-12T17:57:02.383 INFO:tasks.ceph.mon.c:Restarting daemon 2021-02-12T17:57:02.383 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2021-02-12T17:57:02.385 INFO:tasks.ceph.mon.c:Started 2021-02-12T17:57:02.386 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph config set mgr mgr/devicehealth/enable_monitoring false --force 2021-02-12T17:57:14.011 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2021-02-12T17:57:14.011 INFO:tasks.ceph.mgr.x:Restarting daemon 2021-02-12T17:57:14.012 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2021-02-12T17:57:14.014 INFO:tasks.ceph.mgr.x:Started 2021-02-12T17:57:14.014 INFO:tasks.ceph:Setting crush tunables to default 2021-02-12T17:57:14.015 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd crush tunables default 2021-02-12T17:57:14.233 INFO:teuthology.orchestra.run.smithi107.stderr:adjusted tunables profile to default 2021-02-12T17:57:14.247 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2021-02-12T17:57:14.247 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:57:14.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-0/fsid of=/dev/stdout 2021-02-12T17:57:14.271 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:57:14.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-1/fsid of=/dev/stdout 2021-02-12T17:57:14.336 DEBUG:teuthology.orchestra.run.smithi107:> set -ex 2021-02-12T17:57:14.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo dd if=/var/lib/ceph/osd/ceph-2/fsid of=/dev/stdout 2021-02-12T17:57:14.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new c10bf242-c6be-4d2a-817c-901bb5e8f07a 0 2021-02-12T17:57:14.664 INFO:teuthology.orchestra.run.smithi107.stdout:0 2021-02-12T17:57:14.675 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d 1 2021-02-12T17:57:14.896 INFO:teuthology.orchestra.run.smithi107.stdout:1 2021-02-12T17:57:14.909 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd new ab2b9420-4808-48df-b9c1-8683e619c87a 2 2021-02-12T17:57:15.677 INFO:teuthology.orchestra.run.smithi107.stdout:2 2021-02-12T17:57:15.690 INFO:tasks.ceph.osd.0:Restarting daemon 2021-02-12T17:57:15.690 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-02-12T17:57:15.692 INFO:tasks.ceph.osd.0:Started 2021-02-12T17:57:15.692 INFO:tasks.ceph.osd.1:Restarting daemon 2021-02-12T17:57:15.692 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-02-12T17:57:15.695 INFO:tasks.ceph.osd.1:Started 2021-02-12T17:57:15.695 INFO:tasks.ceph.osd.2:Restarting daemon 2021-02-12T17:57:15.695 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2021-02-12T17:57:15.698 INFO:tasks.ceph.osd.2:Started 2021-02-12T17:57:15.698 INFO:tasks.ceph:Waiting for OSDs to come up 2021-02-12T17:57:16.000 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-12T17:57:16.210 INFO:teuthology.misc.health.smithi107.stdout: 2021-02-12T17:57:16.210 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":5,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T17:57:15.124392+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T17:57:16.221 DEBUG:teuthology.misc:0 of 3 OSDs are up 2021-02-12T17:57:16.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:57:16.287+0000 7fb9063d6080 -1 Falling back to public interface 2021-02-12T17:57:16.288 INFO:tasks.ceph.osd.0.smithi107.stderr:2021-02-12T17:57:16.288+0000 7f9b05d49080 -1 Falling back to public interface 2021-02-12T17:57:16.296 INFO:tasks.ceph.osd.2.smithi107.stderr:2021-02-12T17:57:16.295+0000 7f3c25aa5080 -1 Falling back to public interface 2021-02-12T17:57:19.742 INFO:tasks.ceph.osd.0.smithi107.stderr:2021-02-12T17:57:19.741+0000 7f9b05d49080 -1 osd.0 0 log_to_monitors {default=true} 2021-02-12T17:57:19.761 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:57:19.760+0000 7fb9063d6080 -1 osd.1 0 log_to_monitors {default=true} 2021-02-12T17:57:19.768 INFO:tasks.ceph.osd.2.smithi107.stderr:2021-02-12T17:57:19.767+0000 7f3c25aa5080 -1 osd.2 0 log_to_monitors {default=true} 2021-02-12T17:57:21.408 INFO:tasks.ceph.osd.2.smithi107.stderr:2021-02-12T17:57:21.408+0000 7f3c1c22d700 -1 osd.2 0 waiting for initial osdmap 2021-02-12T17:57:21.408 INFO:tasks.ceph.osd.0.smithi107.stderr:2021-02-12T17:57:21.408+0000 7f9afc4d1700 -1 osd.0 0 waiting for initial osdmap 2021-02-12T17:57:21.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:57:21.416+0000 7fb8fcb5e700 -1 osd.1 0 waiting for initial osdmap 2021-02-12T17:57:22.524 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2021-02-12T17:57:22.874 INFO:teuthology.misc.health.smithi107.stdout: 2021-02-12T17:57:22.874 INFO:teuthology.misc.health.smithi107.stdout:{"epoch":8,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T17:57:22.417820+0000","last_up_change":"2021-02-12T17:57:22.417820+0000","last_in_change":"2021-02-12T17:57:22.417820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":33986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":33986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":33986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":33986}]},"public_addr":"172.21.15.107:6802/33986","cluster_addr":"172.21.15.107:6804/33986","heartbeat_back_addr":"172.21.15.107:6807/33986","heartbeat_front_addr":"172.21.15.107:6806/33986","state":["exists","up"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":33987}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":33987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":33987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":33987}]},"public_addr":"172.21.15.107:6800/33987","cluster_addr":"172.21.15.107:6801/33987","heartbeat_back_addr":"172.21.15.107:6805/33987","heartbeat_front_addr":"172.21.15.107:6803/33987","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T17:57:22.886 DEBUG:teuthology.misc:3 of 3 OSDs are up 2021-02-12T17:57:22.886 INFO:tasks.ceph:Creating RBD pool 2021-02-12T17:57:22.886 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool create rbd 8 2021-02-12T17:57:23.482 INFO:teuthology.orchestra.run.smithi107.stderr:pool 'rbd' created 2021-02-12T17:57:23.492 DEBUG:teuthology.orchestra.run.smithi107:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2021-02-12T17:57:24.490 INFO:teuthology.orchestra.run.smithi107.stderr:enabled application 'rbd' on pool 'rbd' 2021-02-12T17:57:24.504 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2021-02-12T17:57:24.504 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-02-12T17:57:24.505 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2021-02-12T17:57:24.802 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:24.802 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":10,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T17:57:24.474885+0000","last_up_change":"2021-02-12T17:57:22.417820+0000","last_in_change":"2021-02-12T17:57:22.417820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-12T17:57:23.210299+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"10","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":33986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":33986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":33986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":33986}]},"public_addr":"172.21.15.107:6802/33986","cluster_addr":"172.21.15.107:6804/33986","heartbeat_back_addr":"172.21.15.107:6807/33986","heartbeat_front_addr":"172.21.15.107:6806/33986","state":["exists","up"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":33987}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":33987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":33987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":33987}]},"public_addr":"172.21.15.107:6800/33987","cluster_addr":"172.21.15.107:6801/33987","heartbeat_back_addr":"172.21.15.107:6805/33987","heartbeat_front_addr":"172.21.15.107:6803/33987","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.700614+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.801932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.752083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T17:57:24.815 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-12T17:57:23.210299+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '10', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-12T17:57:24.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-02-12T17:57:25.109 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2021-02-12T17:57:25.122 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2021-02-12T17:57:25.122 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2021-02-12T17:57:25.122 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2021-02-12T17:57:25.458 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:25.469 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":4,"active_gid":4107,"active_name":"x","active_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6812","nonce":33693}]},"active_addr":"172.21.15.107:6812/33693","active_change":"2021-02-12T17:57:18.391496+0000","active_mgr_features":4540138303579357183,"available":true,"standbys":[],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"docker.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"docker.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"haproxy","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"arcts/keepalived","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"docker.io/prom/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":1645875347}]},{"addrvec":[{"type":"v2","addr":"172.21.15.107:0","nonce":3408038294}]}]}} 2021-02-12T17:57:25.471 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2021-02-12T17:57:25.471 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2021-02-12T17:57:25.472 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-02-12T17:57:25.767 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:25.767 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":11,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T17:57:25.481197+0000","last_up_change":"2021-02-12T17:57:22.417820+0000","last_in_change":"2021-02-12T17:57:22.417820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-12T17:57:23.210299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":33986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":33986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":33986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":33986}]},"public_addr":"172.21.15.107:6802/33986","cluster_addr":"172.21.15.107:6804/33986","heartbeat_back_addr":"172.21.15.107:6807/33986","heartbeat_front_addr":"172.21.15.107:6806/33986","state":["exists","up"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":33987}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":33987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":33987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":33987}]},"public_addr":"172.21.15.107:6800/33987","cluster_addr":"172.21.15.107:6801/33987","heartbeat_back_addr":"172.21.15.107:6805/33987","heartbeat_front_addr":"172.21.15.107:6803/33987","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.700614+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.801932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.752083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T17:57:25.779 INFO:tasks.ceph.ceph_manager.ceph:all up! 2021-02-12T17:57:25.780 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-02-12T17:57:26.071 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:26.071 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":11,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T17:57:25.481197+0000","last_up_change":"2021-02-12T17:57:22.417820+0000","last_in_change":"2021-02-12T17:57:22.417820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-12T17:57:23.210299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":33986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":33986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":33986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":33986}]},"public_addr":"172.21.15.107:6802/33986","cluster_addr":"172.21.15.107:6804/33986","heartbeat_back_addr":"172.21.15.107:6807/33986","heartbeat_front_addr":"172.21.15.107:6806/33986","state":["exists","up"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":33987}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":33987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":33987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":33987}]},"public_addr":"172.21.15.107:6800/33987","cluster_addr":"172.21.15.107:6801/33987","heartbeat_back_addr":"172.21.15.107:6805/33987","heartbeat_front_addr":"172.21.15.107:6803/33987","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.700614+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.801932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.752083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T17:57:26.084 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-02-12T17:57:26.233 INFO:teuthology.orchestra.run.smithi107.stdout:34359738372 2021-02-12T17:57:26.233 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-02-12T17:57:26.383 INFO:teuthology.orchestra.run.smithi107.stdout:34359738372 2021-02-12T17:57:26.384 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-02-12T17:57:26.534 INFO:teuthology.orchestra.run.smithi107.stdout:34359738370 2021-02-12T17:57:26.535 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-02-12T17:57:26.827 INFO:teuthology.orchestra.run.smithi107.stdout:34359738372 2021-02-12T17:57:26.840 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.0 2021-02-12T17:57:26.840 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-12T17:57:27.134 INFO:teuthology.orchestra.run.smithi107.stdout:34359738372 2021-02-12T17:57:27.147 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738372 got 34359738372 for osd.1 2021-02-12T17:57:27.147 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-12T17:57:27.439 INFO:teuthology.orchestra.run.smithi107.stdout:34359738369 2021-02-12T17:57:27.452 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738369 for osd.2 2021-02-12T17:57:28.453 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-12T17:57:28.757 INFO:teuthology.orchestra.run.smithi107.stdout:34359738370 2021-02-12T17:57:28.770 INFO:tasks.ceph.ceph_manager.ceph:need seq 34359738370 got 34359738370 for osd.2 2021-02-12T17:57:28.770 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2021-02-12T17:57:28.770 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:29.058 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:29.060 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:29.070 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-02-12T17:57:28.397464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14712,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":283100808,"statfs":{"total":289910292480,"available":289895227392,"internally_reserved":0,"allocated":294912,"data_stored":66834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.499959+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:24.499959+0000","last_peered":"2021-02-12T17:57:24.499959+0000","last_clean":"2021-02-12T17:57:24.499959+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:24.499959+0000","last_undegraded":"2021-02-12T17:57:24.499959+0000","last_fullsized":"2021-02-12T17:57:24.499959+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.502745+0000","last_change":"2021-02-12T17:57:24.497914+0000","last_active":"2021-02-12T17:57:24.502745+0000","last_peered":"2021-02-12T17:57:24.502745+0000","last_clean":"2021-02-12T17:57:24.502745+0000","last_became_active":"2021-02-12T17:57:24.497629+0000","last_became_peered":"2021-02-12T17:57:24.497629+0000","last_unstale":"2021-02-12T17:57:24.502745+0000","last_undegraded":"2021-02-12T17:57:24.502745+0000","last_fullsized":"2021-02-12T17:57:24.502745+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.494660+0000","last_change":"2021-02-12T17:57:24.490490+0000","last_active":"2021-02-12T17:57:24.494660+0000","last_peered":"2021-02-12T17:57:24.494660+0000","last_clean":"2021-02-12T17:57:24.494660+0000","last_became_active":"2021-02-12T17:57:24.490243+0000","last_became_peered":"2021-02-12T17:57:24.490243+0000","last_unstale":"2021-02-12T17:57:24.494660+0000","last_undegraded":"2021-02-12T17:57:24.494660+0000","last_fullsized":"2021-02-12T17:57:24.494660+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497393+0000","last_change":"2021-02-12T17:57:24.492853+0000","last_active":"2021-02-12T17:57:24.497393+0000","last_peered":"2021-02-12T17:57:24.497393+0000","last_clean":"2021-02-12T17:57:24.497393+0000","last_became_active":"2021-02-12T17:57:24.492514+0000","last_became_peered":"2021-02-12T17:57:24.492514+0000","last_unstale":"2021-02-12T17:57:24.497393+0000","last_undegraded":"2021-02-12T17:57:24.497393+0000","last_fullsized":"2021-02-12T17:57:24.497393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497459+0000","last_change":"2021-02-12T17:57:24.492495+0000","last_active":"2021-02-12T17:57:24.497459+0000","last_peered":"2021-02-12T17:57:24.497459+0000","last_clean":"2021-02-12T17:57:24.497459+0000","last_became_active":"2021-02-12T17:57:24.492269+0000","last_became_peered":"2021-02-12T17:57:24.492269+0000","last_unstale":"2021-02-12T17:57:24.497459+0000","last_undegraded":"2021-02-12T17:57:24.497459+0000","last_fullsized":"2021-02-12T17:57:24.497459+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497419+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:24.497419+0000","last_peered":"2021-02-12T17:57:24.497419+0000","last_clean":"2021-02-12T17:57:24.497419+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:24.497419+0000","last_undegraded":"2021-02-12T17:57:24.497419+0000","last_fullsized":"2021-02-12T17:57:24.497419+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.502766+0000","last_change":"2021-02-12T17:57:24.497921+0000","last_active":"2021-02-12T17:57:24.502766+0000","last_peered":"2021-02-12T17:57:24.502766+0000","last_clean":"2021-02-12T17:57:24.502766+0000","last_became_active":"2021-02-12T17:57:24.497604+0000","last_became_peered":"2021-02-12T17:57:24.497604+0000","last_unstale":"2021-02-12T17:57:24.502766+0000","last_undegraded":"2021-02-12T17:57:24.502766+0000","last_fullsized":"2021-02-12T17:57:24.502766+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.494657+0000","last_change":"2021-02-12T17:57:24.490423+0000","last_active":"2021-02-12T17:57:24.494657+0000","last_peered":"2021-02-12T17:57:24.494657+0000","last_clean":"2021-02-12T17:57:24.494657+0000","last_became_active":"2021-02-12T17:57:24.490156+0000","last_became_peered":"2021-02-12T17:57:24.490156+0000","last_unstale":"2021-02-12T17:57:24.494657+0000","last_undegraded":"2021-02-12T17:57:24.494657+0000","last_fullsized":"2021-02-12T17:57:24.494657+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]}]}} 2021-02-12T17:57:29.071 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:29.358 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:29.360 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:29.371 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":12,"stamp":"2021-02-12T17:57:28.397464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":12,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14712,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":283100808,"statfs":{"total":289910292480,"available":289895227392,"internally_reserved":0,"allocated":294912,"data_stored":66834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":7,"apply_latency_ms":7,"commit_latency_ns":7000000,"apply_latency_ns":7000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.499959+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:24.499959+0000","last_peered":"2021-02-12T17:57:24.499959+0000","last_clean":"2021-02-12T17:57:24.499959+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:24.499959+0000","last_undegraded":"2021-02-12T17:57:24.499959+0000","last_fullsized":"2021-02-12T17:57:24.499959+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.502745+0000","last_change":"2021-02-12T17:57:24.497914+0000","last_active":"2021-02-12T17:57:24.502745+0000","last_peered":"2021-02-12T17:57:24.502745+0000","last_clean":"2021-02-12T17:57:24.502745+0000","last_became_active":"2021-02-12T17:57:24.497629+0000","last_became_peered":"2021-02-12T17:57:24.497629+0000","last_unstale":"2021-02-12T17:57:24.502745+0000","last_undegraded":"2021-02-12T17:57:24.502745+0000","last_fullsized":"2021-02-12T17:57:24.502745+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.494660+0000","last_change":"2021-02-12T17:57:24.490490+0000","last_active":"2021-02-12T17:57:24.494660+0000","last_peered":"2021-02-12T17:57:24.494660+0000","last_clean":"2021-02-12T17:57:24.494660+0000","last_became_active":"2021-02-12T17:57:24.490243+0000","last_became_peered":"2021-02-12T17:57:24.490243+0000","last_unstale":"2021-02-12T17:57:24.494660+0000","last_undegraded":"2021-02-12T17:57:24.494660+0000","last_fullsized":"2021-02-12T17:57:24.494660+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497393+0000","last_change":"2021-02-12T17:57:24.492853+0000","last_active":"2021-02-12T17:57:24.497393+0000","last_peered":"2021-02-12T17:57:24.497393+0000","last_clean":"2021-02-12T17:57:24.497393+0000","last_became_active":"2021-02-12T17:57:24.492514+0000","last_became_peered":"2021-02-12T17:57:24.492514+0000","last_unstale":"2021-02-12T17:57:24.497393+0000","last_undegraded":"2021-02-12T17:57:24.497393+0000","last_fullsized":"2021-02-12T17:57:24.497393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497459+0000","last_change":"2021-02-12T17:57:24.492495+0000","last_active":"2021-02-12T17:57:24.497459+0000","last_peered":"2021-02-12T17:57:24.497459+0000","last_clean":"2021-02-12T17:57:24.497459+0000","last_became_active":"2021-02-12T17:57:24.492269+0000","last_became_peered":"2021-02-12T17:57:24.492269+0000","last_unstale":"2021-02-12T17:57:24.497459+0000","last_undegraded":"2021-02-12T17:57:24.497459+0000","last_fullsized":"2021-02-12T17:57:24.497459+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497419+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:24.497419+0000","last_peered":"2021-02-12T17:57:24.497419+0000","last_clean":"2021-02-12T17:57:24.497419+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:24.497419+0000","last_undegraded":"2021-02-12T17:57:24.497419+0000","last_fullsized":"2021-02-12T17:57:24.497419+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.502766+0000","last_change":"2021-02-12T17:57:24.497921+0000","last_active":"2021-02-12T17:57:24.502766+0000","last_peered":"2021-02-12T17:57:24.502766+0000","last_clean":"2021-02-12T17:57:24.502766+0000","last_became_active":"2021-02-12T17:57:24.497604+0000","last_became_peered":"2021-02-12T17:57:24.497604+0000","last_unstale":"2021-02-12T17:57:24.502766+0000","last_undegraded":"2021-02-12T17:57:24.502766+0000","last_fullsized":"2021-02-12T17:57:24.502766+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.494657+0000","last_change":"2021-02-12T17:57:24.490423+0000","last_active":"2021-02-12T17:57:24.494657+0000","last_peered":"2021-02-12T17:57:24.494657+0000","last_clean":"2021-02-12T17:57:24.494657+0000","last_became_active":"2021-02-12T17:57:24.490156+0000","last_became_peered":"2021-02-12T17:57:24.490156+0000","last_unstale":"2021-02-12T17:57:24.494657+0000","last_undegraded":"2021-02-12T17:57:24.494657+0000","last_fullsized":"2021-02-12T17:57:24.494657+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738370,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},{"osd":0,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53700000000000003}]}]},{"osd":1,"up_from":8,"seq":34359738372,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.502}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52700000000000002}]}]}]}} 2021-02-12T17:57:29.371 INFO:tasks.ceph.ceph_manager.ceph:clean! 2021-02-12T17:57:29.371 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2021-02-12T17:57:29.372 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2021-02-12T17:57:29.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2021-02-12T17:57:29.731 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:29.743 INFO:teuthology.orchestra.run.smithi107.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2021-02-12T17:57:29.744 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2021-02-12T17:57:29.744 INFO:teuthology.run_tasks:Running task osd_backfill... 2021-02-12T17:57:29.757 INFO:tasks.osd_backfill:num_osds is 3 2021-02-12T17:57:29.757 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-02-12T17:57:30.051 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:30.051 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":11,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T17:57:25.481197+0000","last_up_change":"2021-02-12T17:57:22.417820+0000","last_in_change":"2021-02-12T17:57:22.417820+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-12T17:57:23.210299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":33986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":33986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":33986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":33986}]},"public_addr":"172.21.15.107:6802/33986","cluster_addr":"172.21.15.107:6804/33986","heartbeat_back_addr":"172.21.15.107:6807/33986","heartbeat_front_addr":"172.21.15.107:6806/33986","state":["exists","up"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":33987}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":33987}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":33987}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":33987}]},"public_addr":"172.21.15.107:6800/33987","cluster_addr":"172.21.15.107:6801/33987","heartbeat_back_addr":"172.21.15.107:6805/33987","heartbeat_front_addr":"172.21.15.107:6803/33987","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.700614+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.801932+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.752083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T17:57:30.063 INFO:tasks.osd_backfill.ceph_manager:[{'pool': 1, 'pool_name': 'rbd', 'create_time': '2021-02-12T17:57:23.210299+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '11', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2021-02-12T17:57:30.063 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2021-02-12T17:57:30.359 INFO:teuthology.orchestra.run.smithi107.stdout:pg_num: 8 2021-02-12T17:57:30.372 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:epoch 11 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:fsid 9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:created 2021-02-12T17:57:13.894357+0000 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:modified 2021-02-12T17:57:25.481197+0000 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:flags sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:crush_version 4 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:full_ratio 0.95 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:backfillfull_ratio 0.9 2021-02-12T17:57:30.667 INFO:teuthology.orchestra.run.smithi107.stdout:nearfull_ratio 0.85 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:require_min_compat_client luminous 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:min_compat_client jewel 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:require_osd_release quincy 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:stretch_mode_enabled false 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:pool 1 'rbd' replicated size 2 min_size 1 crush_rule 0 object_hash rjenkins pg_num 8 pgp_num 8 autoscale_mode off last_change 11 flags hashpspool stripe_width 0 application rbd 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:max_osd 3 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:osd.0 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6802/33986 v1:172.21.15.107:6804/33986 exists,up c10bf242-c6be-4d2a-817c-901bb5e8f07a 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:osd.1 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6800/33987 v1:172.21.15.107:6801/33987 exists,up b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d 2021-02-12T17:57:30.668 INFO:teuthology.orchestra.run.smithi107.stdout:osd.2 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6808/33988 v1:172.21.15.107:6809/33988 exists,up ab2b9420-4808-48df-b9c1-8683e619c87a 2021-02-12T17:57:30.679 INFO:tasks.osd_backfill.ceph_manager:['osd.0 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6802/33986 v1:172.21.15.107:6804/33986 exists,up c10bf242-c6be-4d2a-817c-901bb5e8f07a', 'osd.1 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6800/33987 v1:172.21.15.107:6801/33987 exists,up b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d', 'osd.2 up in weight 1 up_from 8 up_thru 9 down_at 0 last_clean_interval [0,0) v1:172.21.15.107:6808/33988 v1:172.21.15.107:6809/33988 exists,up ab2b9420-4808-48df-b9c1-8683e619c87a'] 2021-02-12T17:57:30.680 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2021-02-12T17:57:30.829 INFO:teuthology.orchestra.run.smithi107.stdout:34359738374 2021-02-12T17:57:30.829 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-02-12T17:57:30.979 INFO:teuthology.orchestra.run.smithi107.stdout:34359738375 2021-02-12T17:57:30.979 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-02-12T17:57:31.128 INFO:teuthology.orchestra.run.smithi107.stdout:34359738372 2021-02-12T17:57:31.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-02-12T17:57:31.422 INFO:teuthology.orchestra.run.smithi107.stdout:34359738373 2021-02-12T17:57:31.435 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738374 got 34359738373 for osd.0 2021-02-12T17:57:32.436 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2021-02-12T17:57:32.745 INFO:teuthology.orchestra.run.smithi107.stdout:34359738374 2021-02-12T17:57:32.758 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738374 got 34359738374 for osd.0 2021-02-12T17:57:32.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-12T17:57:33.053 INFO:teuthology.orchestra.run.smithi107.stdout:34359738375 2021-02-12T17:57:33.066 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738375 got 34359738375 for osd.1 2021-02-12T17:57:33.066 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-12T17:57:33.359 INFO:teuthology.orchestra.run.smithi107.stdout:34359738372 2021-02-12T17:57:33.371 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738372 got 34359738372 for osd.2 2021-02-12T17:57:33.371 INFO:tasks.osd_backfill.ceph_manager:waiting for clean 2021-02-12T17:57:33.371 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:33.664 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:33.665 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:33.675 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-02-12T17:57:32.398052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14712,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":283100808,"statfs":{"total":289910292480,"available":289895227392,"internally_reserved":0,"allocated":294912,"data_stored":66834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.499959+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:24.499959+0000","last_peered":"2021-02-12T17:57:24.499959+0000","last_clean":"2021-02-12T17:57:24.499959+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:24.499959+0000","last_undegraded":"2021-02-12T17:57:24.499959+0000","last_fullsized":"2021-02-12T17:57:24.499959+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.502745+0000","last_change":"2021-02-12T17:57:24.497914+0000","last_active":"2021-02-12T17:57:24.502745+0000","last_peered":"2021-02-12T17:57:24.502745+0000","last_clean":"2021-02-12T17:57:24.502745+0000","last_became_active":"2021-02-12T17:57:24.497629+0000","last_became_peered":"2021-02-12T17:57:24.497629+0000","last_unstale":"2021-02-12T17:57:24.502745+0000","last_undegraded":"2021-02-12T17:57:24.502745+0000","last_fullsized":"2021-02-12T17:57:24.502745+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.494660+0000","last_change":"2021-02-12T17:57:24.490490+0000","last_active":"2021-02-12T17:57:24.494660+0000","last_peered":"2021-02-12T17:57:24.494660+0000","last_clean":"2021-02-12T17:57:24.494660+0000","last_became_active":"2021-02-12T17:57:24.490243+0000","last_became_peered":"2021-02-12T17:57:24.490243+0000","last_unstale":"2021-02-12T17:57:24.494660+0000","last_undegraded":"2021-02-12T17:57:24.494660+0000","last_fullsized":"2021-02-12T17:57:24.494660+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497393+0000","last_change":"2021-02-12T17:57:24.492853+0000","last_active":"2021-02-12T17:57:24.497393+0000","last_peered":"2021-02-12T17:57:24.497393+0000","last_clean":"2021-02-12T17:57:24.497393+0000","last_became_active":"2021-02-12T17:57:24.492514+0000","last_became_peered":"2021-02-12T17:57:24.492514+0000","last_unstale":"2021-02-12T17:57:24.497393+0000","last_undegraded":"2021-02-12T17:57:24.497393+0000","last_fullsized":"2021-02-12T17:57:24.497393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497459+0000","last_change":"2021-02-12T17:57:24.492495+0000","last_active":"2021-02-12T17:57:24.497459+0000","last_peered":"2021-02-12T17:57:24.497459+0000","last_clean":"2021-02-12T17:57:24.497459+0000","last_became_active":"2021-02-12T17:57:24.492269+0000","last_became_peered":"2021-02-12T17:57:24.492269+0000","last_unstale":"2021-02-12T17:57:24.497459+0000","last_undegraded":"2021-02-12T17:57:24.497459+0000","last_fullsized":"2021-02-12T17:57:24.497459+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497419+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:24.497419+0000","last_peered":"2021-02-12T17:57:24.497419+0000","last_clean":"2021-02-12T17:57:24.497419+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:24.497419+0000","last_undegraded":"2021-02-12T17:57:24.497419+0000","last_fullsized":"2021-02-12T17:57:24.497419+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.502766+0000","last_change":"2021-02-12T17:57:24.497921+0000","last_active":"2021-02-12T17:57:24.502766+0000","last_peered":"2021-02-12T17:57:24.502766+0000","last_clean":"2021-02-12T17:57:24.502766+0000","last_became_active":"2021-02-12T17:57:24.497604+0000","last_became_peered":"2021-02-12T17:57:24.497604+0000","last_unstale":"2021-02-12T17:57:24.502766+0000","last_undegraded":"2021-02-12T17:57:24.502766+0000","last_fullsized":"2021-02-12T17:57:24.502766+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.494657+0000","last_change":"2021-02-12T17:57:24.490423+0000","last_active":"2021-02-12T17:57:24.494657+0000","last_peered":"2021-02-12T17:57:24.494657+0000","last_clean":"2021-02-12T17:57:24.494657+0000","last_became_active":"2021-02-12T17:57:24.490156+0000","last_became_peered":"2021-02-12T17:57:24.490156+0000","last_unstale":"2021-02-12T17:57:24.494657+0000","last_undegraded":"2021-02-12T17:57:24.494657+0000","last_fullsized":"2021-02-12T17:57:24.494657+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]}]}} 2021-02-12T17:57:33.676 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:33.967 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:33.968 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:33.979 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":14,"stamp":"2021-02-12T17:57:32.398052+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":14712,"kb_used_data":288,"kb_used_omap":0,"kb_used_meta":14400,"kb_avail":283100808,"statfs":{"total":289910292480,"available":289895227392,"internally_reserved":0,"allocated":294912,"data_stored":66834,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":14745600},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"0.000000"},"pg_stats":[{"pgid":"1.7","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.499959+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:24.499959+0000","last_peered":"2021-02-12T17:57:24.499959+0000","last_clean":"2021-02-12T17:57:24.499959+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:24.499959+0000","last_undegraded":"2021-02-12T17:57:24.499959+0000","last_fullsized":"2021-02-12T17:57:24.499959+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.502745+0000","last_change":"2021-02-12T17:57:24.497914+0000","last_active":"2021-02-12T17:57:24.502745+0000","last_peered":"2021-02-12T17:57:24.502745+0000","last_clean":"2021-02-12T17:57:24.502745+0000","last_became_active":"2021-02-12T17:57:24.497629+0000","last_became_peered":"2021-02-12T17:57:24.497629+0000","last_unstale":"2021-02-12T17:57:24.502745+0000","last_undegraded":"2021-02-12T17:57:24.502745+0000","last_fullsized":"2021-02-12T17:57:24.502745+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.494660+0000","last_change":"2021-02-12T17:57:24.490490+0000","last_active":"2021-02-12T17:57:24.494660+0000","last_peered":"2021-02-12T17:57:24.494660+0000","last_clean":"2021-02-12T17:57:24.494660+0000","last_became_active":"2021-02-12T17:57:24.490243+0000","last_became_peered":"2021-02-12T17:57:24.490243+0000","last_unstale":"2021-02-12T17:57:24.494660+0000","last_undegraded":"2021-02-12T17:57:24.494660+0000","last_fullsized":"2021-02-12T17:57:24.494660+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497393+0000","last_change":"2021-02-12T17:57:24.492853+0000","last_active":"2021-02-12T17:57:24.497393+0000","last_peered":"2021-02-12T17:57:24.497393+0000","last_clean":"2021-02-12T17:57:24.497393+0000","last_became_active":"2021-02-12T17:57:24.492514+0000","last_became_peered":"2021-02-12T17:57:24.492514+0000","last_unstale":"2021-02-12T17:57:24.497393+0000","last_undegraded":"2021-02-12T17:57:24.497393+0000","last_fullsized":"2021-02-12T17:57:24.497393+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497459+0000","last_change":"2021-02-12T17:57:24.492495+0000","last_active":"2021-02-12T17:57:24.497459+0000","last_peered":"2021-02-12T17:57:24.497459+0000","last_clean":"2021-02-12T17:57:24.497459+0000","last_became_active":"2021-02-12T17:57:24.492269+0000","last_became_peered":"2021-02-12T17:57:24.492269+0000","last_unstale":"2021-02-12T17:57:24.497459+0000","last_undegraded":"2021-02-12T17:57:24.497459+0000","last_fullsized":"2021-02-12T17:57:24.497459+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"1.3","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.497419+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:24.497419+0000","last_peered":"2021-02-12T17:57:24.497419+0000","last_clean":"2021-02-12T17:57:24.497419+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:24.497419+0000","last_undegraded":"2021-02-12T17:57:24.497419+0000","last_fullsized":"2021-02-12T17:57:24.497419+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.502766+0000","last_change":"2021-02-12T17:57:24.497921+0000","last_active":"2021-02-12T17:57:24.502766+0000","last_peered":"2021-02-12T17:57:24.502766+0000","last_clean":"2021-02-12T17:57:24.502766+0000","last_became_active":"2021-02-12T17:57:24.497604+0000","last_became_peered":"2021-02-12T17:57:24.497604+0000","last_unstale":"2021-02-12T17:57:24.502766+0000","last_undegraded":"2021-02-12T17:57:24.502766+0000","last_fullsized":"2021-02-12T17:57:24.502766+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"0'0","reported_seq":"9","reported_epoch":"10","state":"active+clean","last_fresh":"2021-02-12T17:57:24.494657+0000","last_change":"2021-02-12T17:57:24.490423+0000","last_active":"2021-02-12T17:57:24.494657+0000","last_peered":"2021-02-12T17:57:24.494657+0000","last_clean":"2021-02-12T17:57:24.494657+0000","last_became_active":"2021-02-12T17:57:24.490156+0000","last_became_peered":"2021-02-12T17:57:24.490156+0000","last_unstale":"2021-02-12T17:57:24.494657+0000","last_undegraded":"2021-02-12T17:57:24.494657+0000","last_fullsized":"2021-02-12T17:57:24.494657+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,0],"acting":[2,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738372,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45400000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51400000000000001}]}]},{"osd":0,"up_from":8,"seq":34359738374,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47699999999999998}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]}]},{"osd":1,"up_from":8,"seq":34359738375,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":4904,"kb_used_data":96,"kb_used_omap":0,"kb_used_meta":4800,"kb_avail":94366936,"statfs":{"total":96636764160,"available":96631742464,"internally_reserved":0,"allocated":98304,"data_stored":22278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":4915200},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.53500000000000003}]}]}]}} 2021-02-12T17:57:33.980 INFO:tasks.osd_backfill.ceph_manager:clean! 2021-02-12T17:57:33.980 INFO:tasks.osd_backfill:rados -p rbd bench 15 write -b 4096 --no-cleanup 2021-02-12T17:57:33.980 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd bench 15 write -b 4096 --no-cleanup 2021-02-12T17:57:34.021 INFO:teuthology.orchestra.run.smithi107.stdout:hints = 1 2021-02-12T17:57:34.021 INFO:teuthology.orchestra.run.smithi107.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 15 seconds or 0 objects 2021-02-12T17:57:34.022 INFO:teuthology.orchestra.run.smithi107.stdout:Object prefix: benchmark_data_smithi107_37839 2021-02-12T17:57:34.022 INFO:teuthology.orchestra.run.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-02-12T17:57:34.022 INFO:teuthology.orchestra.run.smithi107.stdout: 0 0 0 0 0 0 - 0 2021-02-12T17:57:34.067 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.067+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cba300 legacy=0x558177cbc780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.079 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.078+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.149 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.149+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cbf020 legacy=0x558177cc14a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.149 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.149+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:34.152 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.152+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cba300 legacy=0x7fe05003a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.199 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.198+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe05003a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.584 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.583+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cbf020 legacy=0x7fe050043de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.667 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.667+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.672 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.672+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cbf020 legacy=0x7fe050043de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.736 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.735+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cba300 legacy=0x7fe05003a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:34.890 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.890+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe05003a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:34.891 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.890+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:34.913 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:34.912+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cbf020 legacy=0x7fe050043de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:35.011 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.011+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cbf020 legacy=0x7fe050043de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.022 INFO:teuthology.orchestra.run.smithi107.stdout: 1 16 1837 1821 7.10882 7.11328 0.00944446 0.00874232 2021-02-12T17:57:35.178 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.177+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.231 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.231+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.361 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.360+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cbf020 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.385 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.384+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.399 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.397+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cba300 legacy=0x7fe05003a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.441 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.439+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.554 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.552+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.618 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.616+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:35.779 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.777+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cbf020 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.784 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.782+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.796 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.794+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cbf020 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:35.810 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:35.808+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:36.022 INFO:teuthology.orchestra.run.smithi107.stdout: 2 16 3379 3363 6.56605 6.02344 0.00940555 0.00950025 2021-02-12T17:57:36.050 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.049+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:36.186 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.185+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.200 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.199+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cba300 legacy=0x7fe0500471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.213 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.212+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.233 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.231+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:36.293 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.291+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cba300 legacy=0x7fe0500471e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.310 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.309+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.310+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:36.405 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.404+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cc5cd0 legacy=0x7fe05003a4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.438 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.437+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050044df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.665 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.664+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.731 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.729+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe0500455b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.735 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.734+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.956 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.954+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:36.989 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:36.987+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:37.022 INFO:teuthology.orchestra.run.smithi107.stdout: 3 16 5234 5218 6.79254 7.24609 0.00790627 0.00918767 2021-02-12T17:57:37.091 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:37.089+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:37.178 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:37.176+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:37.190 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:37.188+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cba300 legacy=0x7fe050045d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:37.773 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:37.771+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050045d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.022 INFO:teuthology.orchestra.run.smithi107.stdout: 4 16 7033 7017 6.8511 7.02734 0.00819993 0.00910852 2021-02-12T17:57:38.112 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.110+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.134 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.132+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.159 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.157+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.247 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.245+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cbf020 legacy=0x7fe0500468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.359 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.358+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cba300 legacy=0x7fe050045d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:38.370 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.368+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050044ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.462 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.461+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.469 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.467+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cbf020 legacy=0x7fe0500468e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.483 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.481+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050045ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.816 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.814+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cbf020 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:38.842 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:38.841+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050045ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:39.023 INFO:teuthology.orchestra.run.smithi107.stdout: 5 16 8785 8769 6.84915 6.84375 0.00857006 0.00911651 2021-02-12T17:57:39.085 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.083+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cbf020 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:39.198 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.197+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:39.311 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.310+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cbf020 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:39.442 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.440+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:39.495 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.494+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cbf020 legacy=0x7fe050039520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:39.516 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.515+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:39.756 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.754+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cc5cd0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:39.803 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.801+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:39.811 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.810+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe0500450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:39.845 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.843+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cbf020 legacy=0x7fe050045cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:39.977 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:39.976+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe0500450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:40.023 INFO:teuthology.orchestra.run.smithi107.stdout: 6 16 10457 10441 6.79609 6.53125 0.00875949 0.00918465 2021-02-12T17:57:40.161 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:40.159+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cbf020 legacy=0x7fe0500458d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:40.415 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:40.413+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:40.443 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:40.439+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe0500450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:40.484 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:40.483+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cbf020 legacy=0x7fe0500398f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:40.736 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:40.735+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe0500450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:40.748 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:40.746+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:40.936 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:40.935+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:41.023 INFO:teuthology.orchestra.run.smithi107.stdout: 7 16 12054 12038 6.71635 6.23828 0.00470313 0.00929583 2021-02-12T17:57:41.538 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.536+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:41.549 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.548+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe05003a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.722+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe050044c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:41.724 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.722+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe050044c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:41.762 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.761+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:41.813 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.811+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:41.867 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.866+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:41.891 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.890+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:41.916 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:41.914+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:42.023 INFO:teuthology.orchestra.run.smithi107.stdout: 8 16 13572 13556 6.61777 5.92969 0.0106496 0.00943504 2021-02-12T17:57:42.162 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:42.161+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:42.975 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:42.973+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.023 INFO:teuthology.orchestra.run.smithi107.stdout: 9 16 15057 15041 6.52695 5.80078 0.00862519 0.0095672 2021-02-12T17:57:43.072 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.070+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.254 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.253+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.326 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.325+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe050044c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.391 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.390+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.427 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.425+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.560 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.558+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.601 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.599+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe0500398f0 legacy=0x7fe050044c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.684 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.682+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:43.706 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.704+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.715 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.713+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.829 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.826+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe050044ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:43.848 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.847+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:43.928 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:43.924+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:44.023 INFO:teuthology.orchestra.run.smithi107.stdout: 10 16 16465 16449 6.42402 5.5 0.0131676 0.0097196 2021-02-12T17:57:44.128 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:44.127+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe0500398f0 legacy=0x7fe050044ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:44.148 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:44.146+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:44.239 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:44.237+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe050044ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:44.578 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:44.576+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe050044b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:44.854 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:44.853+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:44.989 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:44.987+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.023 INFO:teuthology.orchestra.run.smithi107.stdout: 11 16 17827 17811 6.32366 5.32031 0.00790143 0.00987516 2021-02-12T17:57:45.157 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.155+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.179 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.177+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.264 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.262+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.326 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.325+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.632 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.630+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:45.667 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.666+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.759 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.758+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:45.851 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.849+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.910 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.909+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.975 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.973+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:45.977 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:45.976+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:46.024 INFO:teuthology.orchestra.run.smithi107.stdout: 12 16 18894 18878 6.14402 4.16797 0.00690754 0.0101635 2021-02-12T17:57:46.069 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.068+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:46.082 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.079+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:46.435 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.434+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:46.436 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.434+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:46.505 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.504+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:46.553 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.551+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:46.558 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.557+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:46.829 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.828+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:46.888 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.887+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:46.932 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:46.931+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:47.024 INFO:teuthology.orchestra.run.smithi107.stdout: 13 16 20184 20168 6.059 5.03906 0.0178651 0.0103074 2021-02-12T17:57:47.027 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.026+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.056 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.054+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.089 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.087+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.100 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.098+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:47.104 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.102+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.126 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.125+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.286 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.284+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.370 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.368+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.806 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.804+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.856 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.855+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:47.941 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:47.939+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:48.024 INFO:teuthology.orchestra.run.smithi107.stdout: 14 16 21432 21416 5.97441 4.875 0.0136251 0.010454 2021-02-12T17:57:48.075 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:48.073+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe05003a480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:48.075 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:48.074+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:57:48.161 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:48.159+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe0500398f0 legacy=0x7fe050047450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:48.184 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:48.183+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x558177cba300 legacy=0x7fe050046890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:48.227 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:48.226+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:48.311 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:48.309+0000 7fe06d684700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6808/33988 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:48.870 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:48.868+0000 7fe06de85700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6800/33987 conn(0x7fe050035850 legacy=0x7fe050037c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:48.871 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:57:48.870+0000 7fe06ce83700 0 -- 172.21.15.107:0/1097032857 >> v1:172.21.15.107:6802/33986 conn(0x558177cc5cd0 legacy=0x7fe050039cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:57:49.025 INFO:teuthology.orchestra.run.smithi107.stdout: 15 14 22627 22613 5.88784 4.67578 0.00977334 0.0106089 2021-02-12T17:57:50.024 INFO:teuthology.orchestra.run.smithi107.stdout:Total time run: 15.0106 2021-02-12T17:57:50.024 INFO:teuthology.orchestra.run.smithi107.stdout:Total writes made: 22627 2021-02-12T17:57:50.024 INFO:teuthology.orchestra.run.smithi107.stdout:Write size: 4096 2021-02-12T17:57:50.024 INFO:teuthology.orchestra.run.smithi107.stdout:Object size: 4096 2021-02-12T17:57:50.024 INFO:teuthology.orchestra.run.smithi107.stdout:Bandwidth (MB/sec): 5.88828 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Stddev Bandwidth: 0.955028 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Max bandwidth (MB/sec): 7.24609 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Min bandwidth (MB/sec): 4.16797 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Average IOPS: 1507 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Stddev IOPS: 244.487 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Max IOPS: 1855 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Min IOPS: 1067 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Average Latency(s): 0.0106085 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Stddev Latency(s): 0.00806585 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Max latency(s): 0.217306 2021-02-12T17:57:50.025 INFO:teuthology.orchestra.run.smithi107.stdout:Min latency(s): 0.00185592 2021-02-12T17:57:50.030 INFO:tasks.osd_backfill:err is 0 2021-02-12T17:57:50.031 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd out 0 2021-02-12T17:57:50.392 INFO:tasks.ceph.mon.a.smithi107.stderr:2021-02-12T17:57:50.391+0000 7f0fd92a8700 -1 mon.a@0(leader).osd e11 definitely_dead 0 2021-02-12T17:57:50.580 INFO:teuthology.orchestra.run.smithi107.stderr:marked out osd.0. 2021-02-12T17:57:50.591 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2021-02-12T17:57:50.591 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T17:57:50.627 INFO:tasks.ceph.osd.0:Stopped 2021-02-12T17:57:50.627 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd down 0 2021-02-12T17:57:50.973 INFO:tasks.ceph.mon.a.smithi107.stderr:2021-02-12T17:57:50.972+0000 7f0fd92a8700 -1 mon.a@0(leader).osd e12 definitely_dead 0 2021-02-12T17:57:51.578 INFO:teuthology.orchestra.run.smithi107.stderr:marked down osd.0. 2021-02-12T17:57:51.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-02-12T17:57:51.744 INFO:teuthology.orchestra.run.smithi107.stdout:34359738381 2021-02-12T17:57:51.745 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-02-12T17:57:51.895 INFO:teuthology.orchestra.run.smithi107.stdout:34359738377 2021-02-12T17:57:51.896 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-12T17:57:52.191 INFO:teuthology.orchestra.run.smithi107.stdout:34359738380 2021-02-12T17:57:52.204 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738381 got 34359738380 for osd.1 2021-02-12T17:57:53.204 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-12T17:57:53.595 INFO:teuthology.orchestra.run.smithi107.stdout:34359738381 2021-02-12T17:57:53.611 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738381 got 34359738381 for osd.1 2021-02-12T17:57:53.611 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-12T17:57:53.975 INFO:teuthology.orchestra.run.smithi107.stdout:34359738377 2021-02-12T17:57:53.990 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738377 got 34359738377 for osd.2 2021-02-12T17:57:53.990 INFO:tasks.osd_backfill.ceph_manager:waiting for recovery to complete 2021-02-12T17:57:53.991 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:54.359 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:54.361 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:54.374 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":27,"stamp":"2021-02-12T17:57:52.576237+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":10,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":161140,"kb_used_data":113636,"kb_used_omap":0,"kb_used_meta":47488,"kb_avail":188582540,"statfs":{"total":193273528320,"available":193108520960,"internally_reserved":0,"allocated":116363264,"data_stored":116184908,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48627712},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":8,"apply_latency_ms":8,"commit_latency_ns":8000000,"apply_latency_ns":8000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28733464,"num_objects":7016,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":28061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.175830"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2919","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-12T17:57:50.571691+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:50.571691+0000","last_peered":"2021-02-12T17:57:50.571691+0000","last_clean":"2021-02-12T17:57:50.571691+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:50.571691+0000","last_undegraded":"2021-02-12T17:57:50.571691+0000","last_fullsized":"2021-02-12T17:57:50.571691+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2847","reported_epoch":"13","state":"remapped+peering","last_fresh":"2021-02-12T17:57:51.573619+0000","last_change":"2021-02-12T17:57:51.573815+0000","last_active":"2021-02-12T17:57:50.573071+0000","last_peered":"2021-02-12T17:57:49.025648+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:24.497629+0000","last_became_peered":"2021-02-12T17:57:24.497629+0000","last_unstale":"2021-02-12T17:57:51.573619+0000","last_undegraded":"2021-02-12T17:57:51.573619+0000","last_fullsized":"2021-02-12T17:57:51.573619+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"2870","reported_epoch":"13","state":"remapped+peering","last_fresh":"2021-02-12T17:57:51.573082+0000","last_change":"2021-02-12T17:57:51.573277+0000","last_active":"2021-02-12T17:57:50.573727+0000","last_peered":"2021-02-12T17:57:49.019427+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:24.490243+0000","last_became_peered":"2021-02-12T17:57:24.490243+0000","last_unstale":"2021-02-12T17:57:51.573082+0000","last_undegraded":"2021-02-12T17:57:51.573082+0000","last_fullsized":"2021-02-12T17:57:51.573082+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2759","reported_epoch":"13","state":"remapped+peering","last_fresh":"2021-02-12T17:57:51.573560+0000","last_change":"2021-02-12T17:57:51.573730+0000","last_active":"2021-02-12T17:57:50.572565+0000","last_peered":"2021-02-12T17:57:49.032475+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:24.492514+0000","last_became_peered":"2021-02-12T17:57:24.492514+0000","last_unstale":"2021-02-12T17:57:51.573560+0000","last_undegraded":"2021-02-12T17:57:51.573560+0000","last_fullsized":"2021-02-12T17:57:51.573560+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"2790","reported_epoch":"13","state":"remapped+peering","last_fresh":"2021-02-12T17:57:51.574190+0000","last_change":"2021-02-12T17:57:51.574366+0000","last_active":"2021-02-12T17:57:50.571871+0000","last_peered":"2021-02-12T17:57:49.012740+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:24.492269+0000","last_became_peered":"2021-02-12T17:57:24.492269+0000","last_unstale":"2021-02-12T17:57:51.574190+0000","last_undegraded":"2021-02-12T17:57:51.574190+0000","last_fullsized":"2021-02-12T17:57:51.574190+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2828","reported_epoch":"12","state":"active+clean","last_fresh":"2021-02-12T17:57:50.571633+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:50.571633+0000","last_peered":"2021-02-12T17:57:50.571633+0000","last_clean":"2021-02-12T17:57:50.571633+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:50.571633+0000","last_undegraded":"2021-02-12T17:57:50.571633+0000","last_fullsized":"2021-02-12T17:57:50.571633+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2844","reported_epoch":"13","state":"remapped+peering","last_fresh":"2021-02-12T17:57:51.573663+0000","last_change":"2021-02-12T17:57:51.573843+0000","last_active":"2021-02-12T17:57:50.572628+0000","last_peered":"2021-02-12T17:57:49.032320+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:24.497604+0000","last_became_peered":"2021-02-12T17:57:24.497604+0000","last_unstale":"2021-02-12T17:57:51.573663+0000","last_undegraded":"2021-02-12T17:57:51.573663+0000","last_fullsized":"2021-02-12T17:57:51.573663+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2869","reported_epoch":"13","state":"remapped+peering","last_fresh":"2021-02-12T17:57:51.573108+0000","last_change":"2021-02-12T17:57:51.573290+0000","last_active":"2021-02-12T17:57:50.572285+0000","last_peered":"2021-02-12T17:57:50.028083+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:24.490156+0000","last_became_peered":"2021-02-12T17:57:24.490156+0000","last_unstale":"2021-02-12T17:57:51.573108+0000","last_undegraded":"2021-02-12T17:57:51.573108+0000","last_fullsized":"2021-02-12T17:57:51.573108+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[1],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183132160,"data_stored":183128088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738377,"num_pgs":4,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":66800,"kb_used_data":45864,"kb_used_omap":0,"kb_used_meta":20928,"kb_avail":94305040,"statfs":{"total":96636764160,"available":96568360960,"internally_reserved":0,"allocated":46964736,"data_stored":46873522,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":21430272},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,0,3,5,3,1],"upper_bound":64},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.8340000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738381,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94340,"kb_used_data":67772,"kb_used_omap":0,"kb_used_meta":26560,"kb_avail":94277500,"statfs":{"total":96636764160,"available":96540160000,"internally_reserved":0,"allocated":69398528,"data_stored":69311386,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27197440},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]}]}} 2021-02-12T17:57:54.376 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:54.759 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:54.760 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:54.773 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2021-02-12T17:57:54.401865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":166616,"kb_used_data":113736,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":188577064,"statfs":{"total":193273528320,"available":193102913536,"internally_reserved":0,"allocated":116465664,"data_stored":116275966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28733464,"num_objects":7016,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":28061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001149"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2920","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573582+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:51.573582+0000","last_peered":"2021-02-12T17:57:51.573582+0000","last_clean":"2021-02-12T17:57:51.573582+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:51.573582+0000","last_undegraded":"2021-02-12T17:57:51.573582+0000","last_fullsized":"2021-02-12T17:57:51.573582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2853","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.727878+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.727878+0000","last_peered":"2021-02-12T17:57:52.727878+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.727878+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"3005","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:53.419797+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:57:53.419797+0000","last_peered":"2021-02-12T17:57:53.419797+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:57:53.419797+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2765","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.721418+0000","last_change":"2021-02-12T17:57:52.722534+0000","last_active":"2021-02-12T17:57:52.721418+0000","last_peered":"2021-02-12T17:57:52.721418+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:57:52.721418+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"2925","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:53.421087+0000","last_change":"2021-02-12T17:57:52.705180+0000","last_active":"2021-02-12T17:57:53.421087+0000","last_peered":"2021-02-12T17:57:53.421087+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:57:53.421087+0000","last_undegraded":"2021-02-12T17:57:52.604412+0000","last_fullsized":"2021-02-12T17:57:52.578020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2829","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573353+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:51.573353+0000","last_peered":"2021-02-12T17:57:51.573353+0000","last_clean":"2021-02-12T17:57:51.573353+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:51.573353+0000","last_undegraded":"2021-02-12T17:57:51.573353+0000","last_fullsized":"2021-02-12T17:57:51.573353+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2850","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728124+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728124+0000","last_peered":"2021-02-12T17:57:52.728124+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728124+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2875","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.702073+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.702073+0000","last_peered":"2021-02-12T17:57:52.702073+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.702073+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183660544,"data_stored":183656472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72204,"kb_used_data":45956,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94299636,"statfs":{"total":96636764160,"available":96562827264,"internally_reserved":0,"allocated":47058944,"data_stored":46962059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94412,"kb_used_data":67780,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94277428,"statfs":{"total":96636764160,"available":96540086272,"internally_reserved":0,"allocated":69406720,"data_stored":69313907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]}]}} 2021-02-12T17:57:54.775 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:55.152 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:55.155 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:55.167 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":28,"stamp":"2021-02-12T17:57:54.401865+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":166616,"kb_used_data":113736,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":188577064,"statfs":{"total":193273528320,"available":193102913536,"internally_reserved":0,"allocated":116465664,"data_stored":116275966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":28733464,"num_objects":7016,"num_object_clones":0,"num_object_copies":14032,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":7016,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":7016,"num_write_kb":28061,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.001149"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2920","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573582+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:51.573582+0000","last_peered":"2021-02-12T17:57:51.573582+0000","last_clean":"2021-02-12T17:57:51.573582+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:51.573582+0000","last_undegraded":"2021-02-12T17:57:51.573582+0000","last_fullsized":"2021-02-12T17:57:51.573582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2853","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.727878+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.727878+0000","last_peered":"2021-02-12T17:57:52.727878+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.727878+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"3005","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:53.419797+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:57:53.419797+0000","last_peered":"2021-02-12T17:57:53.419797+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:57:53.419797+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2765","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.721418+0000","last_change":"2021-02-12T17:57:52.722534+0000","last_active":"2021-02-12T17:57:52.721418+0000","last_peered":"2021-02-12T17:57:52.721418+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:57:52.721418+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"2925","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:53.421087+0000","last_change":"2021-02-12T17:57:52.705180+0000","last_active":"2021-02-12T17:57:53.421087+0000","last_peered":"2021-02-12T17:57:53.421087+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:57:53.421087+0000","last_undegraded":"2021-02-12T17:57:52.604412+0000","last_fullsized":"2021-02-12T17:57:52.578020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2829","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573353+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:51.573353+0000","last_peered":"2021-02-12T17:57:51.573353+0000","last_clean":"2021-02-12T17:57:51.573353+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:51.573353+0000","last_undegraded":"2021-02-12T17:57:51.573353+0000","last_fullsized":"2021-02-12T17:57:51.573353+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2850","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728124+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728124+0000","last_peered":"2021-02-12T17:57:52.728124+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728124+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2875","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.702073+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.702073+0000","last_peered":"2021-02-12T17:57:52.702073+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.702073+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183660544,"data_stored":183656472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72204,"kb_used_data":45956,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94299636,"statfs":{"total":96636764160,"available":96562827264,"internally_reserved":0,"allocated":47058944,"data_stored":46962059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94412,"kb_used_data":67780,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94277428,"statfs":{"total":96636764160,"available":96540086272,"internally_reserved":0,"allocated":69406720,"data_stored":69313907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]}]}} 2021-02-12T17:57:58.169 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:59.497 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:59.499 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:59.512 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2021-02-12T17:57:58.402580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":166616,"kb_used_data":113736,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":188577064,"statfs":{"total":193273528320,"available":193102913536,"internally_reserved":0,"allocated":116465664,"data_stored":116275966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2977816,"num_objects":728,"num_object_clones":0,"num_object_copies":1456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":728,"num_write_kb":2909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.824622"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2920","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573582+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:51.573582+0000","last_peered":"2021-02-12T17:57:51.573582+0000","last_clean":"2021-02-12T17:57:51.573582+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:51.573582+0000","last_undegraded":"2021-02-12T17:57:51.573582+0000","last_fullsized":"2021-02-12T17:57:51.573582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2853","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.727878+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.727878+0000","last_peered":"2021-02-12T17:57:52.727878+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.727878+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"3005","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:53.419797+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:57:53.419797+0000","last_peered":"2021-02-12T17:57:53.419797+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:57:53.419797+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2765","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.721418+0000","last_change":"2021-02-12T17:57:52.722534+0000","last_active":"2021-02-12T17:57:52.721418+0000","last_peered":"2021-02-12T17:57:52.721418+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:57:52.721418+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"2925","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:53.421087+0000","last_change":"2021-02-12T17:57:52.705180+0000","last_active":"2021-02-12T17:57:53.421087+0000","last_peered":"2021-02-12T17:57:53.421087+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:57:53.421087+0000","last_undegraded":"2021-02-12T17:57:52.604412+0000","last_fullsized":"2021-02-12T17:57:52.578020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2829","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573353+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:51.573353+0000","last_peered":"2021-02-12T17:57:51.573353+0000","last_clean":"2021-02-12T17:57:51.573353+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:51.573353+0000","last_undegraded":"2021-02-12T17:57:51.573353+0000","last_fullsized":"2021-02-12T17:57:51.573353+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2850","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728124+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728124+0000","last_peered":"2021-02-12T17:57:52.728124+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728124+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2875","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.702073+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.702073+0000","last_peered":"2021-02-12T17:57:52.702073+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.702073+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183660544,"data_stored":183656472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72204,"kb_used_data":45956,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94299636,"statfs":{"total":96636764160,"available":96562827264,"internally_reserved":0,"allocated":47058944,"data_stored":46962059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94412,"kb_used_data":67780,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94277428,"statfs":{"total":96636764160,"available":96540086272,"internally_reserved":0,"allocated":69406720,"data_stored":69313907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]}]}} 2021-02-12T17:57:59.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:57:59.886 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:57:59.888 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:57:59.902 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":30,"stamp":"2021-02-12T17:57:58.402580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":14,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":166616,"kb_used_data":113736,"kb_used_omap":0,"kb_used_meta":52864,"kb_avail":188577064,"statfs":{"total":193273528320,"available":193102913536,"internally_reserved":0,"allocated":116465664,"data_stored":116275966,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":54132736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":2977816,"num_objects":728,"num_object_clones":0,"num_object_copies":1456,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":728,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":728,"num_write_kb":2909,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.824622"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2920","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573582+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:51.573582+0000","last_peered":"2021-02-12T17:57:51.573582+0000","last_clean":"2021-02-12T17:57:51.573582+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:51.573582+0000","last_undegraded":"2021-02-12T17:57:51.573582+0000","last_fullsized":"2021-02-12T17:57:51.573582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2853","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.727878+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.727878+0000","last_peered":"2021-02-12T17:57:52.727878+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.727878+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"3005","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:53.419797+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:57:53.419797+0000","last_peered":"2021-02-12T17:57:53.419797+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:57:53.419797+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2793,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":64}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2765","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.721418+0000","last_change":"2021-02-12T17:57:52.722534+0000","last_active":"2021-02-12T17:57:52.721418+0000","last_peered":"2021-02-12T17:57:52.721418+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:57:52.721418+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"2925","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:53.421087+0000","last_change":"2021-02-12T17:57:52.705180+0000","last_active":"2021-02-12T17:57:53.421087+0000","last_peered":"2021-02-12T17:57:53.421087+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:57:53.421087+0000","last_undegraded":"2021-02-12T17:57:52.604412+0000","last_fullsized":"2021-02-12T17:57:52.578020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2715,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":64,"num_bytes_recovered":262144,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":64}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2829","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573353+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:51.573353+0000","last_peered":"2021-02-12T17:57:51.573353+0000","last_clean":"2021-02-12T17:57:51.573353+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:51.573353+0000","last_undegraded":"2021-02-12T17:57:51.573353+0000","last_fullsized":"2021-02-12T17:57:51.573353+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2850","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728124+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728124+0000","last_peered":"2021-02-12T17:57:52.728124+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728124+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2875","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.702073+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.702073+0000","last_peered":"2021-02-12T17:57:52.702073+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.702073+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":16775,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":128,"num_bytes_recovered":524288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":183660544,"data_stored":183656472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738378,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":72204,"kb_used_data":45956,"kb_used_omap":0,"kb_used_meta":26240,"kb_avail":94299636,"statfs":{"total":96636764160,"available":96562827264,"internally_reserved":0,"allocated":47058944,"data_stored":46962059,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":26869760},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41199999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738382,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":94412,"kb_used_data":67780,"kb_used_omap":0,"kb_used_meta":26624,"kb_avail":94277428,"statfs":{"total":96636764160,"available":96540086272,"internally_reserved":0,"allocated":69406720,"data_stored":69313907,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27262976},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46000000000000002}]}]}]}} 2021-02-12T17:58:02.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:03.288 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:03.290 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:03.302 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2021-02-12T17:58:02.403303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":6008832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":184772,"kb_used_data":118900,"kb_used_omap":0,"kb_used_meta":65856,"kb_avail":188558908,"statfs":{"total":193273528320,"available":193084321792,"internally_reserved":0,"allocated":121753600,"data_stored":121563902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67436544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":6008832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001746"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2920","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573582+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:51.573582+0000","last_peered":"2021-02-12T17:57:51.573582+0000","last_clean":"2021-02-12T17:57:51.573582+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:51.573582+0000","last_undegraded":"2021-02-12T17:57:51.573582+0000","last_fullsized":"2021-02-12T17:57:51.573582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2853","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.727878+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.727878+0000","last_peered":"2021-02-12T17:57:52.727878+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.727878+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"4305","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:58.417657+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:57:58.417657+0000","last_peered":"2021-02-12T17:57:58.417657+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:57:58.417657+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":714,"num_bytes_recovered":2924544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2765","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.721418+0000","last_change":"2021-02-12T17:57:52.722534+0000","last_active":"2021-02-12T17:57:52.721418+0000","last_peered":"2021-02-12T17:57:52.721418+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:57:52.721418+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"4303","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:58.424566+0000","last_change":"2021-02-12T17:57:52.705180+0000","last_active":"2021-02-12T17:57:58.424566+0000","last_peered":"2021-02-12T17:57:58.424566+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:57:58.424566+0000","last_undegraded":"2021-02-12T17:57:52.604412+0000","last_fullsized":"2021-02-12T17:57:52.578020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":753,"num_bytes_recovered":3084288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":753}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2829","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573353+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:51.573353+0000","last_peered":"2021-02-12T17:57:51.573353+0000","last_clean":"2021-02-12T17:57:51.573353+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:51.573353+0000","last_undegraded":"2021-02-12T17:57:51.573353+0000","last_fullsized":"2021-02-12T17:57:51.573353+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2850","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728124+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728124+0000","last_peered":"2021-02-12T17:57:52.728124+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728124+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2875","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.702073+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.702073+0000","last_peered":"2021-02-12T17:57:52.702073+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.702073+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":6008832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":189140992,"data_stored":189136920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738379,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80016,"kb_used_data":48584,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":94291824,"statfs":{"total":96636764160,"available":96554827776,"internally_reserved":0,"allocated":49750016,"data_stored":49653131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104756,"kb_used_data":70316,"kb_used_omap":0,"kb_used_meta":34432,"kb_avail":94267084,"statfs":{"total":96636764160,"available":96529494016,"internally_reserved":0,"allocated":72003584,"data_stored":71910771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35258368},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]}]}} 2021-02-12T17:58:03.303 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:03.688 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:03.690 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:03.704 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":32,"stamp":"2021-02-12T17:58:02.403303+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":6008832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":184772,"kb_used_data":118900,"kb_used_omap":0,"kb_used_meta":65856,"kb_avail":188558908,"statfs":{"total":193273528320,"available":193084321792,"internally_reserved":0,"allocated":121753600,"data_stored":121563902,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":67436544},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":6008832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001746"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2920","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573582+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:51.573582+0000","last_peered":"2021-02-12T17:57:51.573582+0000","last_clean":"2021-02-12T17:57:51.573582+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:51.573582+0000","last_undegraded":"2021-02-12T17:57:51.573582+0000","last_fullsized":"2021-02-12T17:57:51.573582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2853","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.727878+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.727878+0000","last_peered":"2021-02-12T17:57:52.727878+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.727878+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"4305","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:58.417657+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:57:58.417657+0000","last_peered":"2021-02-12T17:57:58.417657+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:57:58.417657+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2143,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":714,"num_bytes_recovered":2924544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":714}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2765","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.721418+0000","last_change":"2021-02-12T17:57:52.722534+0000","last_active":"2021-02-12T17:57:52.721418+0000","last_peered":"2021-02-12T17:57:52.721418+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:57:52.721418+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"4303","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:58.424566+0000","last_change":"2021-02-12T17:57:52.705180+0000","last_active":"2021-02-12T17:57:58.424566+0000","last_peered":"2021-02-12T17:57:58.424566+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:57:58.424566+0000","last_undegraded":"2021-02-12T17:57:52.604412+0000","last_fullsized":"2021-02-12T17:57:52.578020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2026,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":753,"num_bytes_recovered":3084288,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":753}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2829","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573353+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:51.573353+0000","last_peered":"2021-02-12T17:57:51.573353+0000","last_clean":"2021-02-12T17:57:51.573353+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:51.573353+0000","last_undegraded":"2021-02-12T17:57:51.573353+0000","last_fullsized":"2021-02-12T17:57:51.573353+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2850","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728124+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728124+0000","last_peered":"2021-02-12T17:57:52.728124+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728124+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2875","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.702073+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.702073+0000","last_peered":"2021-02-12T17:57:52.702073+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.702073+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15436,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1467,"num_bytes_recovered":6008832,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":189140992,"data_stored":189136920,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738379,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":80016,"kb_used_data":48584,"kb_used_omap":0,"kb_used_meta":31424,"kb_avail":94291824,"statfs":{"total":96636764160,"available":96554827776,"internally_reserved":0,"allocated":49750016,"data_stored":49653131,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32178176},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.35499999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":104756,"kb_used_data":70316,"kb_used_omap":0,"kb_used_meta":34432,"kb_avail":94267084,"statfs":{"total":96636764160,"available":96529494016,"internally_reserved":0,"allocated":72003584,"data_stored":71910771,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":35258368},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34399999999999997}]}]}]}} 2021-02-12T17:58:06.705 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:07.094 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:07.096 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:07.109 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":34,"stamp":"2021-02-12T17:58:06.404000+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3261,"num_bytes_recovered":13357056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":202684,"kb_used_data":125804,"kb_used_omap":0,"kb_used_meta":76864,"kb_avail":188540996,"statfs":{"total":193273528320,"available":193065979904,"internally_reserved":0,"allocated":128823296,"data_stored":128633598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":78708736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3133,"num_bytes_recovered":12832768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002134"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2920","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573582+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:51.573582+0000","last_peered":"2021-02-12T17:57:51.573582+0000","last_clean":"2021-02-12T17:57:51.573582+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:51.573582+0000","last_undegraded":"2021-02-12T17:57:51.573582+0000","last_fullsized":"2021-02-12T17:57:51.573582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2853","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.727878+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.727878+0000","last_peered":"2021-02-12T17:57:52.727878+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.727878+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"6027","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:03.422063+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:58:03.422063+0000","last_peered":"2021-02-12T17:58:03.422063+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:58:03.422063+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":6451200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1575}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2765","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.721418+0000","last_change":"2021-02-12T17:57:52.722534+0000","last_active":"2021-02-12T17:57:52.721418+0000","last_peered":"2021-02-12T17:57:52.721418+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:57:52.721418+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"6169","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:03.426527+0000","last_change":"2021-02-12T17:57:52.705180+0000","last_active":"2021-02-12T17:58:03.426527+0000","last_peered":"2021-02-12T17:58:03.426527+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:58:03.426527+0000","last_undegraded":"2021-02-12T17:57:52.604412+0000","last_fullsized":"2021-02-12T17:57:52.578020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":6905856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2829","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573353+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:51.573353+0000","last_peered":"2021-02-12T17:57:51.573353+0000","last_clean":"2021-02-12T17:57:51.573353+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:51.573353+0000","last_undegraded":"2021-02-12T17:57:51.573353+0000","last_fullsized":"2021-02-12T17:57:51.573353+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2850","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728124+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728124+0000","last_peered":"2021-02-12T17:57:52.728124+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728124+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2875","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.702073+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.702073+0000","last_peered":"2021-02-12T17:57:52.702073+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.702073+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3261,"num_bytes_recovered":13357056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196489216,"data_stored":196485144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89240,"kb_used_data":52240,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":94282600,"statfs":{"total":96636764160,"available":96545382400,"internally_reserved":0,"allocated":53493760,"data_stored":53396875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113444,"kb_used_data":73564,"kb_used_omap":0,"kb_used_meta":39872,"kb_avail":94258396,"statfs":{"total":96636764160,"available":96520597504,"internally_reserved":0,"allocated":75329536,"data_stored":75236723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40828928},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]}]}} 2021-02-12T17:58:07.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:07.495 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:07.498 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:07.511 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2021-02-12T17:58:07.477467+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3261,"num_bytes_recovered":13357056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":202684,"kb_used_data":125804,"kb_used_omap":0,"kb_used_meta":76864,"kb_avail":188540996,"statfs":{"total":193273528320,"available":193065979904,"internally_reserved":0,"allocated":128823296,"data_stored":128633598,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":78708736},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3133,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3133,"num_bytes_recovered":12832768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"11.075190"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2920","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573582+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:51.573582+0000","last_peered":"2021-02-12T17:57:51.573582+0000","last_clean":"2021-02-12T17:57:51.573582+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:51.573582+0000","last_undegraded":"2021-02-12T17:57:51.573582+0000","last_fullsized":"2021-02-12T17:57:51.573582+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2853","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.727878+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.727878+0000","last_peered":"2021-02-12T17:57:52.727878+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.727878+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"6027","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:03.422063+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:58:03.422063+0000","last_peered":"2021-02-12T17:58:03.422063+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:58:03.422063+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1282,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1575,"num_bytes_recovered":6451200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1575}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"2765","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.721418+0000","last_change":"2021-02-12T17:57:52.722534+0000","last_active":"2021-02-12T17:57:52.721418+0000","last_peered":"2021-02-12T17:57:52.721418+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:57:52.721418+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2746,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2746}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"6169","reported_epoch":"14","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:03.426527+0000","last_change":"2021-02-12T17:57:52.705180+0000","last_active":"2021-02-12T17:58:03.426527+0000","last_peered":"2021-02-12T17:58:03.426527+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:58:03.426527+0000","last_undegraded":"2021-02-12T17:57:52.604412+0000","last_fullsized":"2021-02-12T17:57:52.578020+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1093,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1686,"num_bytes_recovered":6905856,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1686}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2829","reported_epoch":"13","state":"active+clean","last_fresh":"2021-02-12T17:57:51.573353+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:51.573353+0000","last_peered":"2021-02-12T17:57:51.573353+0000","last_clean":"2021-02-12T17:57:51.573353+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:51.573353+0000","last_undegraded":"2021-02-12T17:57:51.573353+0000","last_fullsized":"2021-02-12T17:57:51.573353+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2850","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728124+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728124+0000","last_peered":"2021-02-12T17:57:52.728124+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728124+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2875","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.702073+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.702073+0000","last_peered":"2021-02-12T17:57:52.702073+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.702073+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13642,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3261,"num_bytes_recovered":13357056,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":196489216,"data_stored":196485144,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89240,"kb_used_data":52240,"kb_used_omap":0,"kb_used_meta":36992,"kb_avail":94282600,"statfs":{"total":96636764160,"available":96545382400,"internally_reserved":0,"allocated":53493760,"data_stored":53396875,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":37879808},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40600000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113444,"kb_used_data":73564,"kb_used_omap":0,"kb_used_meta":39872,"kb_avail":94258396,"statfs":{"total":96636764160,"available":96520597504,"internally_reserved":0,"allocated":75329536,"data_stored":75236723,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40828928},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34000000000000002}]}]}]}} 2021-02-12T17:58:10.513 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:10.892 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:10.894 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:10.909 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2021-02-12T17:58:10.485712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5731,"num_bytes_recovered":23474176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":191544,"kb_used_data":135272,"kb_used_omap":0,"kb_used_meta":56256,"kb_avail":188552136,"statfs":{"total":193273528320,"available":193077387264,"internally_reserved":0,"allocated":138518528,"data_stored":138317464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57606144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2470,"num_bytes_recovered":10117120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.081712"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2921","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-12T17:57:52.578237+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:52.578237+0000","last_peered":"2021-02-12T17:57:52.578237+0000","last_clean":"2021-02-12T17:57:52.578237+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:52.578237+0000","last_undegraded":"2021-02-12T17:57:52.578237+0000","last_fullsized":"2021-02-12T17:57:52.578237+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2854","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.729130+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.729130+0000","last_peered":"2021-02-12T17:57:52.729130+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.729130+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8514","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:08.418303+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:58:08.418303+0000","last_peered":"2021-02-12T17:58:08.418303+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:58:08.418303+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2818}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"3035","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:08.425594+0000","last_change":"2021-02-12T17:58:07.451540+0000","last_active":"2021-02-12T17:58:08.425594+0000","last_peered":"2021-02-12T17:58:08.425594+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:08.425594+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":548864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":134}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8358","reported_epoch":"15","state":"peering","last_fresh":"2021-02-12T17:58:07.481560+0000","last_change":"2021-02-12T17:58:07.482041+0000","last_active":"2021-02-12T17:58:07.481560+0000","last_peered":"2021-02-12T17:58:07.450883+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:58:07.481560+0000","last_undegraded":"2021-02-12T17:58:07.481560+0000","last_fullsized":"2021-02-12T17:58:07.481560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2830","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-12T17:57:52.578021+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:52.578021+0000","last_peered":"2021-02-12T17:57:52.578021+0000","last_clean":"2021-02-12T17:57:52.578021+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:52.578021+0000","last_undegraded":"2021-02-12T17:57:52.578021+0000","last_fullsized":"2021-02-12T17:57:52.578021+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2851","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728972+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728972+0000","last_peered":"2021-02-12T17:57:52.728972+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728972+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2876","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.704870+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.704870+0000","last_peered":"2021-02-12T17:57:52.704870+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.704870+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5731,"num_bytes_recovered":23474176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206618624,"data_stored":206614552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84036,"kb_used_data":57212,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94287804,"statfs":{"total":96636764160,"available":96550711296,"internally_reserved":0,"allocated":58585088,"data_stored":58482520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107508,"kb_used_data":78060,"kb_used_omap":0,"kb_used_meta":29440,"kb_avail":94264332,"statfs":{"total":96636764160,"available":96526675968,"internally_reserved":0,"allocated":79933440,"data_stored":79834944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30146560},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]}]}]}} 2021-02-12T17:58:10.910 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:11.300 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:11.303 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:11.315 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":40,"stamp":"2021-02-12T17:58:10.485712+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5731,"num_bytes_recovered":23474176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":191544,"kb_used_data":135272,"kb_used_omap":0,"kb_used_meta":56256,"kb_avail":188552136,"statfs":{"total":193273528320,"available":193077387264,"internally_reserved":0,"allocated":138518528,"data_stored":138317464,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":57606144},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2470,"num_bytes_recovered":10117120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.081712"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2921","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-12T17:57:52.578237+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:57:52.578237+0000","last_peered":"2021-02-12T17:57:52.578237+0000","last_clean":"2021-02-12T17:57:52.578237+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:57:52.578237+0000","last_undegraded":"2021-02-12T17:57:52.578237+0000","last_fullsized":"2021-02-12T17:57:52.578237+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2854","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.729130+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:57:52.729130+0000","last_peered":"2021-02-12T17:57:52.729130+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:57:52.729130+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8514","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:08.418303+0000","last_change":"2021-02-12T17:57:52.706212+0000","last_active":"2021-02-12T17:58:08.418303+0000","last_peered":"2021-02-12T17:58:08.418303+0000","last_clean":"2021-02-12T17:57:49.019427+0000","last_became_active":"2021-02-12T17:57:52.694560+0000","last_became_peered":"2021-02-12T17:57:52.694560+0000","last_unstale":"2021-02-12T17:58:08.418303+0000","last_undegraded":"2021-02-12T17:57:52.594143+0000","last_fullsized":"2021-02-12T17:57:52.577071+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":39,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2818,"num_bytes_recovered":11542528,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2818}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"3035","reported_epoch":"15","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:08.425594+0000","last_change":"2021-02-12T17:58:07.451540+0000","last_active":"2021-02-12T17:58:08.425594+0000","last_peered":"2021-02-12T17:58:08.425594+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:08.425594+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2612,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":134,"num_bytes_recovered":548864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":134}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8358","reported_epoch":"15","state":"peering","last_fresh":"2021-02-12T17:58:07.481560+0000","last_change":"2021-02-12T17:58:07.482041+0000","last_active":"2021-02-12T17:58:07.481560+0000","last_peered":"2021-02-12T17:58:07.450883+0000","last_clean":"2021-02-12T17:57:49.012740+0000","last_became_active":"2021-02-12T17:57:52.696348+0000","last_became_peered":"2021-02-12T17:57:52.696348+0000","last_unstale":"2021-02-12T17:58:07.481560+0000","last_undegraded":"2021-02-12T17:58:07.481560+0000","last_fullsized":"2021-02-12T17:58:07.481560+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2830","reported_epoch":"14","state":"active+clean","last_fresh":"2021-02-12T17:57:52.578021+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:57:52.578021+0000","last_peered":"2021-02-12T17:57:52.578021+0000","last_clean":"2021-02-12T17:57:52.578021+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:57:52.578021+0000","last_undegraded":"2021-02-12T17:57:52.578021+0000","last_fullsized":"2021-02-12T17:57:52.578021+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2851","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.728972+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:57:52.728972+0000","last_peered":"2021-02-12T17:57:52.728972+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:57:52.728972+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"2876","reported_epoch":"14","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:57:52.704870+0000","last_change":"2021-02-12T17:57:52.704870+0000","last_active":"2021-02-12T17:57:52.704870+0000","last_peered":"2021-02-12T17:57:52.704870+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:57:52.704870+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2856,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11172,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5731,"num_bytes_recovered":23474176,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":206618624,"data_stored":206614552,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738381,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":84036,"kb_used_data":57212,"kb_used_omap":0,"kb_used_meta":26816,"kb_avail":94287804,"statfs":{"total":96636764160,"available":96550711296,"internally_reserved":0,"allocated":58585088,"data_stored":58482520,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":27459584},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40799999999999997}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":107508,"kb_used_data":78060,"kb_used_omap":0,"kb_used_meta":29440,"kb_avail":94264332,"statfs":{"total":96636764160,"available":96526675968,"internally_reserved":0,"allocated":79933440,"data_stored":79834944,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":30146560},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]}]}]}} 2021-02-12T17:58:14.317 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:14.695 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:14.697 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:14.709 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2021-02-12T17:58:14.405201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29679616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":213164,"kb_used_data":141852,"kb_used_omap":0,"kb_used_meta":71296,"kb_avail":188530516,"statfs":{"total":193273528320,"available":193055248384,"internally_reserved":0,"allocated":145256448,"data_stored":145021102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73007104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":16322560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000900"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2924","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.491802+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:09.491802+0000","last_peered":"2021-02-12T17:58:09.491802+0000","last_clean":"2021-02-12T17:58:09.491802+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:09.491802+0000","last_undegraded":"2021-02-12T17:58:09.491802+0000","last_fullsized":"2021-02-12T17:58:09.491802+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2857","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490907+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:09.490907+0000","last_peered":"2021-02-12T17:58:09.490907+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:09.490907+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8604","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.496930+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:10.496930+0000","last_peered":"2021-02-12T17:58:10.496930+0000","last_clean":"2021-02-12T17:58:10.496930+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:10.496930+0000","last_undegraded":"2021-02-12T17:58:10.496930+0000","last_fullsized":"2021-02-12T17:58:10.496930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"4512","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:13.425543+0000","last_change":"2021-02-12T17:58:07.451540+0000","last_active":"2021-02-12T17:58:13.425543+0000","last_peered":"2021-02-12T17:58:13.425543+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:13.425543+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":3567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":871}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8368","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489265+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:09.489265+0000","last_peered":"2021-02-12T17:58:09.489265+0000","last_clean":"2021-02-12T17:58:09.489265+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:09.489265+0000","last_undegraded":"2021-02-12T17:58:09.489265+0000","last_fullsized":"2021-02-12T17:58:09.489265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2833","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489264+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:09.489264+0000","last_peered":"2021-02-12T17:58:09.489264+0000","last_clean":"2021-02-12T17:58:09.489264+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:09.489264+0000","last_undegraded":"2021-02-12T17:58:09.489264+0000","last_fullsized":"2021-02-12T17:58:09.489264+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2854","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490574+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:58:09.490574+0000","last_peered":"2021-02-12T17:58:09.490574+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:09.490574+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"4358","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:13.419824+0000","last_change":"2021-02-12T17:58:08.531975+0000","last_active":"2021-02-12T17:58:13.419824+0000","last_peered":"2021-02-12T17:58:13.419824+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:58:13.419824+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":3026944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29679616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212819968,"data_stored":212815896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92276,"kb_used_data":60140,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94279564,"statfs":{"total":96636764160,"available":96542273536,"internally_reserved":0,"allocated":61583360,"data_stored":61463651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120888,"kb_used_data":81712,"kb_used_omap":0,"kb_used_meta":39168,"kb_avail":94250952,"statfs":{"total":96636764160,"available":96512974848,"internally_reserved":0,"allocated":83673088,"data_stored":83557451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40108032},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]}]}} 2021-02-12T17:58:14.710 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:15.093 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:15.096 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:15.108 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":42,"stamp":"2021-02-12T17:58:14.405201+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29679616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":213164,"kb_used_data":141852,"kb_used_omap":0,"kb_used_meta":71296,"kb_avail":188530516,"statfs":{"total":193273528320,"available":193055248384,"internally_reserved":0,"allocated":145256448,"data_stored":145021102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73007104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3985,"num_bytes_recovered":16322560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000900"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2924","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.491802+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:09.491802+0000","last_peered":"2021-02-12T17:58:09.491802+0000","last_clean":"2021-02-12T17:58:09.491802+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:09.491802+0000","last_undegraded":"2021-02-12T17:58:09.491802+0000","last_fullsized":"2021-02-12T17:58:09.491802+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2857","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490907+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:09.490907+0000","last_peered":"2021-02-12T17:58:09.490907+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:09.490907+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8604","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.496930+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:10.496930+0000","last_peered":"2021-02-12T17:58:10.496930+0000","last_clean":"2021-02-12T17:58:10.496930+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:10.496930+0000","last_undegraded":"2021-02-12T17:58:10.496930+0000","last_fullsized":"2021-02-12T17:58:10.496930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"4512","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:13.425543+0000","last_change":"2021-02-12T17:58:07.451540+0000","last_active":"2021-02-12T17:58:13.425543+0000","last_peered":"2021-02-12T17:58:13.425543+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:13.425543+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":3567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":871}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8368","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489265+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:09.489265+0000","last_peered":"2021-02-12T17:58:09.489265+0000","last_clean":"2021-02-12T17:58:09.489265+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:09.489265+0000","last_undegraded":"2021-02-12T17:58:09.489265+0000","last_fullsized":"2021-02-12T17:58:09.489265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2833","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489264+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:09.489264+0000","last_peered":"2021-02-12T17:58:09.489264+0000","last_clean":"2021-02-12T17:58:09.489264+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:09.489264+0000","last_undegraded":"2021-02-12T17:58:09.489264+0000","last_fullsized":"2021-02-12T17:58:09.489264+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2854","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490574+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:58:09.490574+0000","last_peered":"2021-02-12T17:58:09.490574+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:09.490574+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"4358","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:13.419824+0000","last_change":"2021-02-12T17:58:08.531975+0000","last_active":"2021-02-12T17:58:13.419824+0000","last_peered":"2021-02-12T17:58:13.419824+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:58:13.419824+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":3026944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29679616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212819968,"data_stored":212815896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92276,"kb_used_data":60140,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94279564,"statfs":{"total":96636764160,"available":96542273536,"internally_reserved":0,"allocated":61583360,"data_stored":61463651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120888,"kb_used_data":81712,"kb_used_omap":0,"kb_used_meta":39168,"kb_avail":94250952,"statfs":{"total":96636764160,"available":96512974848,"internally_reserved":0,"allocated":83673088,"data_stored":83557451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40108032},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]}]}} 2021-02-12T17:58:18.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:19.187 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:19.189 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:19.201 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2021-02-12T17:58:18.405874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29679616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":213164,"kb_used_data":141852,"kb_used_omap":0,"kb_used_meta":71296,"kb_avail":188530516,"statfs":{"total":193273528320,"available":193055248384,"internally_reserved":0,"allocated":145256448,"data_stored":145021102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73007104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1515,"num_bytes_recovered":6205440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.918475"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2924","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.491802+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:09.491802+0000","last_peered":"2021-02-12T17:58:09.491802+0000","last_clean":"2021-02-12T17:58:09.491802+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:09.491802+0000","last_undegraded":"2021-02-12T17:58:09.491802+0000","last_fullsized":"2021-02-12T17:58:09.491802+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2857","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490907+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:09.490907+0000","last_peered":"2021-02-12T17:58:09.490907+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:09.490907+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8604","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.496930+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:10.496930+0000","last_peered":"2021-02-12T17:58:10.496930+0000","last_clean":"2021-02-12T17:58:10.496930+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:10.496930+0000","last_undegraded":"2021-02-12T17:58:10.496930+0000","last_fullsized":"2021-02-12T17:58:10.496930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"4512","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:13.425543+0000","last_change":"2021-02-12T17:58:07.451540+0000","last_active":"2021-02-12T17:58:13.425543+0000","last_peered":"2021-02-12T17:58:13.425543+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:13.425543+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":3567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":871}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8368","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489265+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:09.489265+0000","last_peered":"2021-02-12T17:58:09.489265+0000","last_clean":"2021-02-12T17:58:09.489265+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:09.489265+0000","last_undegraded":"2021-02-12T17:58:09.489265+0000","last_fullsized":"2021-02-12T17:58:09.489265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2833","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489264+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:09.489264+0000","last_peered":"2021-02-12T17:58:09.489264+0000","last_clean":"2021-02-12T17:58:09.489264+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:09.489264+0000","last_undegraded":"2021-02-12T17:58:09.489264+0000","last_fullsized":"2021-02-12T17:58:09.489264+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2854","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490574+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:58:09.490574+0000","last_peered":"2021-02-12T17:58:09.490574+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:09.490574+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"4358","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:13.419824+0000","last_change":"2021-02-12T17:58:08.531975+0000","last_active":"2021-02-12T17:58:13.419824+0000","last_peered":"2021-02-12T17:58:13.419824+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:58:13.419824+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":3026944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29679616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212819968,"data_stored":212815896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92276,"kb_used_data":60140,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94279564,"statfs":{"total":96636764160,"available":96542273536,"internally_reserved":0,"allocated":61583360,"data_stored":61463651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120888,"kb_used_data":81712,"kb_used_omap":0,"kb_used_meta":39168,"kb_avail":94250952,"statfs":{"total":96636764160,"available":96512974848,"internally_reserved":0,"allocated":83673088,"data_stored":83557451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40108032},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]}]}} 2021-02-12T17:58:19.202 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:19.590 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:19.593 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:19.605 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":44,"stamp":"2021-02-12T17:58:18.405874+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29679616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":213164,"kb_used_data":141852,"kb_used_omap":0,"kb_used_meta":71296,"kb_avail":188530516,"statfs":{"total":193273528320,"available":193055248384,"internally_reserved":0,"allocated":145256448,"data_stored":145021102,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":73007104},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1515,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1515,"num_bytes_recovered":6205440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.918475"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2924","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.491802+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:09.491802+0000","last_peered":"2021-02-12T17:58:09.491802+0000","last_clean":"2021-02-12T17:58:09.491802+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:09.491802+0000","last_undegraded":"2021-02-12T17:58:09.491802+0000","last_fullsized":"2021-02-12T17:58:09.491802+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2857","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490907+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:09.490907+0000","last_peered":"2021-02-12T17:58:09.490907+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:09.490907+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8604","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.496930+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:10.496930+0000","last_peered":"2021-02-12T17:58:10.496930+0000","last_clean":"2021-02-12T17:58:10.496930+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:10.496930+0000","last_undegraded":"2021-02-12T17:58:10.496930+0000","last_fullsized":"2021-02-12T17:58:10.496930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"4512","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:13.425543+0000","last_change":"2021-02-12T17:58:07.451540+0000","last_active":"2021-02-12T17:58:13.425543+0000","last_peered":"2021-02-12T17:58:13.425543+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:13.425543+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1875,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":871,"num_bytes_recovered":3567616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":871}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8368","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489265+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:09.489265+0000","last_peered":"2021-02-12T17:58:09.489265+0000","last_clean":"2021-02-12T17:58:09.489265+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:09.489265+0000","last_undegraded":"2021-02-12T17:58:09.489265+0000","last_fullsized":"2021-02-12T17:58:09.489265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2833","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489264+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:09.489264+0000","last_peered":"2021-02-12T17:58:09.489264+0000","last_clean":"2021-02-12T17:58:09.489264+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:09.489264+0000","last_undegraded":"2021-02-12T17:58:09.489264+0000","last_fullsized":"2021-02-12T17:58:09.489264+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2854","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490574+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:58:09.490574+0000","last_peered":"2021-02-12T17:58:09.490574+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:09.490574+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"4358","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:13.419824+0000","last_change":"2021-02-12T17:58:08.531975+0000","last_active":"2021-02-12T17:58:13.419824+0000","last_peered":"2021-02-12T17:58:13.419824+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:58:13.419824+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":739,"num_bytes_recovered":3026944,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":739}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9657,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":7246,"num_bytes_recovered":29679616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":212819968,"data_stored":212815896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738382,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":92276,"kb_used_data":60140,"kb_used_omap":0,"kb_used_meta":32128,"kb_avail":94279564,"statfs":{"total":96636764160,"available":96542273536,"internally_reserved":0,"allocated":61583360,"data_stored":61463651,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":32899072},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46700000000000003}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":120888,"kb_used_data":81712,"kb_used_omap":0,"kb_used_meta":39168,"kb_avail":94250952,"statfs":{"total":96636764160,"available":96512974848,"internally_reserved":0,"allocated":83673088,"data_stored":83557451,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40108032},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.29799999999999999}]}]}]}} 2021-02-12T17:58:22.607 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:23.002 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:23.003 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:23.016 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2021-02-12T17:58:22.478101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9008,"num_bytes_recovered":36896768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":218808,"kb_used_data":148776,"kb_used_omap":0,"kb_used_meta":70016,"kb_avail":188524872,"statfs":{"total":193273528320,"available":193049468928,"internally_reserved":0,"allocated":152346624,"data_stored":152111278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71696384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3277,"num_bytes_recovered":13422592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.073208"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2924","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.491802+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:09.491802+0000","last_peered":"2021-02-12T17:58:09.491802+0000","last_clean":"2021-02-12T17:58:09.491802+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:09.491802+0000","last_undegraded":"2021-02-12T17:58:09.491802+0000","last_fullsized":"2021-02-12T17:58:09.491802+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2857","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490907+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:09.490907+0000","last_peered":"2021-02-12T17:58:09.490907+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:09.490907+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8604","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.496930+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:10.496930+0000","last_peered":"2021-02-12T17:58:10.496930+0000","last_clean":"2021-02-12T17:58:10.496930+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:10.496930+0000","last_undegraded":"2021-02-12T17:58:10.496930+0000","last_fullsized":"2021-02-12T17:58:10.496930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"6350","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:18.428679+0000","last_change":"2021-02-12T17:58:07.451540+0000","last_active":"2021-02-12T17:58:18.428679+0000","last_peered":"2021-02-12T17:58:18.428679+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:18.428679+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1790,"num_bytes_recovered":7331840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1790}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8368","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489265+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:09.489265+0000","last_peered":"2021-02-12T17:58:09.489265+0000","last_clean":"2021-02-12T17:58:09.489265+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:09.489265+0000","last_undegraded":"2021-02-12T17:58:09.489265+0000","last_fullsized":"2021-02-12T17:58:09.489265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2833","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489264+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:09.489264+0000","last_peered":"2021-02-12T17:58:09.489264+0000","last_clean":"2021-02-12T17:58:09.489264+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:09.489264+0000","last_undegraded":"2021-02-12T17:58:09.489264+0000","last_fullsized":"2021-02-12T17:58:09.489264+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2854","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490574+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:58:09.490574+0000","last_peered":"2021-02-12T17:58:09.490574+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:09.490574+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"6044","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:18.420757+0000","last_change":"2021-02-12T17:58:08.531975+0000","last_active":"2021-02-12T17:58:18.420757+0000","last_peered":"2021-02-12T17:58:18.420757+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:58:18.420757+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1582,"num_bytes_recovered":6479872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9008,"num_bytes_recovered":36896768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":220033024,"data_stored":220028952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101476,"kb_used_data":63772,"kb_used_omap":0,"kb_used_meta":37696,"kb_avail":94270364,"statfs":{"total":96636764160,"available":96532852736,"internally_reserved":0,"allocated":65302528,"data_stored":65182819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38600704},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117332,"kb_used_data":85004,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":94254508,"statfs":{"total":96636764160,"available":96516616192,"internally_reserved":0,"allocated":87044096,"data_stored":86928459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]}]}]}} 2021-02-12T17:58:23.017 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:23.362 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:23.364 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:23.376 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":47,"stamp":"2021-02-12T17:58:22.478101+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9008,"num_bytes_recovered":36896768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":218808,"kb_used_data":148776,"kb_used_omap":0,"kb_used_meta":70016,"kb_avail":188524872,"statfs":{"total":193273528320,"available":193049468928,"internally_reserved":0,"allocated":152346624,"data_stored":152111278,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":71696384},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3277,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3277,"num_bytes_recovered":13422592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.073208"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2924","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.491802+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:09.491802+0000","last_peered":"2021-02-12T17:58:09.491802+0000","last_clean":"2021-02-12T17:58:09.491802+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:09.491802+0000","last_undegraded":"2021-02-12T17:58:09.491802+0000","last_fullsized":"2021-02-12T17:58:09.491802+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2857","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490907+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:09.490907+0000","last_peered":"2021-02-12T17:58:09.490907+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:09.490907+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8604","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.496930+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:10.496930+0000","last_peered":"2021-02-12T17:58:10.496930+0000","last_clean":"2021-02-12T17:58:10.496930+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:10.496930+0000","last_undegraded":"2021-02-12T17:58:10.496930+0000","last_fullsized":"2021-02-12T17:58:10.496930+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"6350","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:18.428679+0000","last_change":"2021-02-12T17:58:07.451540+0000","last_active":"2021-02-12T17:58:18.428679+0000","last_peered":"2021-02-12T17:58:18.428679+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:18.428679+0000","last_undegraded":"2021-02-12T17:57:52.604956+0000","last_fullsized":"2021-02-12T17:57:52.578019+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":956,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1790,"num_bytes_recovered":7331840,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1790}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8368","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489265+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:09.489265+0000","last_peered":"2021-02-12T17:58:09.489265+0000","last_clean":"2021-02-12T17:58:09.489265+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:09.489265+0000","last_undegraded":"2021-02-12T17:58:09.489265+0000","last_fullsized":"2021-02-12T17:58:09.489265+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2833","reported_epoch":"17","state":"active+clean","last_fresh":"2021-02-12T17:58:09.489264+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:09.489264+0000","last_peered":"2021-02-12T17:58:09.489264+0000","last_clean":"2021-02-12T17:58:09.489264+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:09.489264+0000","last_undegraded":"2021-02-12T17:58:09.489264+0000","last_fullsized":"2021-02-12T17:58:09.489264+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"2854","reported_epoch":"17","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:09.490574+0000","last_change":"2021-02-12T17:57:52.728972+0000","last_active":"2021-02-12T17:58:09.490574+0000","last_peered":"2021-02-12T17:58:09.490574+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:09.490574+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2831,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2831}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"6044","reported_epoch":"18","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:18.420757+0000","last_change":"2021-02-12T17:58:08.531975+0000","last_active":"2021-02-12T17:58:18.420757+0000","last_peered":"2021-02-12T17:58:18.420757+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:58:18.420757+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1274,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1582,"num_bytes_recovered":6479872,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":1582}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7895,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":9008,"num_bytes_recovered":36896768,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":220033024,"data_stored":220028952,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738383,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":101476,"kb_used_data":63772,"kb_used_omap":0,"kb_used_meta":37696,"kb_avail":94270364,"statfs":{"total":96636764160,"available":96532852736,"internally_reserved":0,"allocated":65302528,"data_stored":65182819,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38600704},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41599999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":117332,"kb_used_data":85004,"kb_used_omap":0,"kb_used_meta":32320,"kb_avail":94254508,"statfs":{"total":96636764160,"available":96516616192,"internally_reserved":0,"allocated":87044096,"data_stored":86928459,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":33095680},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.40500000000000003}]}]}]}} 2021-02-12T17:58:26.377 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:26.725 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:26.727 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:26.739 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2021-02-12T17:58:26.407134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11478,"num_bytes_recovered":47009816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":245496,"kb_used_data":158952,"kb_used_omap":0,"kb_used_meta":86528,"kb_avail":188498184,"statfs":{"total":193273528320,"available":193022140416,"internally_reserved":0,"allocated":162766848,"data_stored":162520036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88604672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2470,"num_bytes_recovered":10113048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000935"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2925","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.493504+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:10.493504+0000","last_peered":"2021-02-12T17:58:10.493504+0000","last_clean":"2021-02-12T17:58:10.493504+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:10.493504+0000","last_undegraded":"2021-02-12T17:58:10.493504+0000","last_fullsized":"2021-02-12T17:58:10.493504+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2858","reported_epoch":"18","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:10.492589+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:10.492589+0000","last_peered":"2021-02-12T17:58:10.492589+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:10.492589+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8605","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.498200+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:10.498200+0000","last_peered":"2021-02-12T17:58:10.498200+0000","last_clean":"2021-02-12T17:58:10.498200+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:10.498200+0000","last_undegraded":"2021-02-12T17:58:10.498200+0000","last_fullsized":"2021-02-12T17:58:10.498200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8265","reported_epoch":"19","state":"peering","last_fresh":"2021-02-12T17:58:22.480857+0000","last_change":"2021-02-12T17:58:22.481220+0000","last_active":"2021-02-12T17:58:22.480857+0000","last_peered":"2021-02-12T17:58:21.756513+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:22.480857+0000","last_undegraded":"2021-02-12T17:58:22.480857+0000","last_fullsized":"2021-02-12T17:58:22.480857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8369","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.489281+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:10.489281+0000","last_peered":"2021-02-12T17:58:10.489281+0000","last_clean":"2021-02-12T17:58:10.489281+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:10.489281+0000","last_undegraded":"2021-02-12T17:58:10.489281+0000","last_fullsized":"2021-02-12T17:58:10.489281+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2834","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.489315+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:10.489315+0000","last_peered":"2021-02-12T17:58:10.489315+0000","last_clean":"2021-02-12T17:58:10.489315+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:10.489315+0000","last_undegraded":"2021-02-12T17:58:10.489315+0000","last_fullsized":"2021-02-12T17:58:10.489315+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"3336","reported_epoch":"19","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:23.430492+0000","last_change":"2021-02-12T17:58:21.756987+0000","last_active":"2021-02-12T17:58:23.430492+0000","last_peered":"2021-02-12T17:58:23.430492+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:23.430492+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":240}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8594","reported_epoch":"19","state":"active+undersized+remapped","last_fresh":"2021-02-12T17:58:23.150126+0000","last_change":"2021-02-12T17:58:23.150400+0000","last_active":"2021-02-12T17:58:23.150126+0000","last_peered":"2021-02-12T17:58:23.150126+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:58:23.150126+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11478,"num_bytes_recovered":47009816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":230141952,"data_stored":230133808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112632,"kb_used_data":68784,"kb_used_omap":0,"kb_used_meta":43840,"kb_avail":94259208,"statfs":{"total":96636764160,"available":96521428992,"internally_reserved":0,"allocated":70434816,"data_stored":70309374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44892160},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132864,"kb_used_data":90168,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":94238976,"statfs":{"total":96636764160,"available":96500711424,"internally_reserved":0,"allocated":92332032,"data_stored":92210662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]}]}} 2021-02-12T17:58:26.740 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:27.075 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:27.078 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:27.089 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":51,"stamp":"2021-02-12T17:58:26.407134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11478,"num_bytes_recovered":47009816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":245496,"kb_used_data":158952,"kb_used_omap":0,"kb_used_meta":86528,"kb_avail":188498184,"statfs":{"total":193273528320,"available":193022140416,"internally_reserved":0,"allocated":162766848,"data_stored":162520036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88604672},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2470,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2470,"num_bytes_recovered":10113048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.000935"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2925","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.493504+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:10.493504+0000","last_peered":"2021-02-12T17:58:10.493504+0000","last_clean":"2021-02-12T17:58:10.493504+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:10.493504+0000","last_undegraded":"2021-02-12T17:58:10.493504+0000","last_fullsized":"2021-02-12T17:58:10.493504+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2858","reported_epoch":"18","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:10.492589+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:10.492589+0000","last_peered":"2021-02-12T17:58:10.492589+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:10.492589+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8605","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.498200+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:10.498200+0000","last_peered":"2021-02-12T17:58:10.498200+0000","last_clean":"2021-02-12T17:58:10.498200+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:10.498200+0000","last_undegraded":"2021-02-12T17:58:10.498200+0000","last_fullsized":"2021-02-12T17:58:10.498200+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8265","reported_epoch":"19","state":"peering","last_fresh":"2021-02-12T17:58:22.480857+0000","last_change":"2021-02-12T17:58:22.481220+0000","last_active":"2021-02-12T17:58:22.480857+0000","last_peered":"2021-02-12T17:58:21.756513+0000","last_clean":"2021-02-12T17:57:49.032475+0000","last_became_active":"2021-02-12T17:57:52.717784+0000","last_became_peered":"2021-02-12T17:57:52.717784+0000","last_unstale":"2021-02-12T17:58:22.480857+0000","last_undegraded":"2021-02-12T17:58:22.480857+0000","last_fullsized":"2021-02-12T17:58:22.480857+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[2],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8369","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.489281+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:10.489281+0000","last_peered":"2021-02-12T17:58:10.489281+0000","last_clean":"2021-02-12T17:58:10.489281+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:10.489281+0000","last_undegraded":"2021-02-12T17:58:10.489281+0000","last_fullsized":"2021-02-12T17:58:10.489281+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2834","reported_epoch":"18","state":"active+clean","last_fresh":"2021-02-12T17:58:10.489315+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:10.489315+0000","last_peered":"2021-02-12T17:58:10.489315+0000","last_clean":"2021-02-12T17:58:10.489315+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:10.489315+0000","last_undegraded":"2021-02-12T17:58:10.489315+0000","last_fullsized":"2021-02-12T17:58:10.489315+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"3336","reported_epoch":"19","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:23.430492+0000","last_change":"2021-02-12T17:58:21.756987+0000","last_active":"2021-02-12T17:58:23.430492+0000","last_peered":"2021-02-12T17:58:23.430492+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:23.430492+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2591,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":240,"num_bytes_recovered":983040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":240}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8594","reported_epoch":"19","state":"active+undersized+remapped","last_fresh":"2021-02-12T17:58:23.150126+0000","last_change":"2021-02-12T17:58:23.150400+0000","last_active":"2021-02-12T17:58:23.150126+0000","last_peered":"2021-02-12T17:58:23.150126+0000","last_clean":"2021-02-12T17:57:50.028083+0000","last_became_active":"2021-02-12T17:57:52.698661+0000","last_became_peered":"2021-02-12T17:57:52.698661+0000","last_unstale":"2021-02-12T17:58:23.150126+0000","last_undegraded":"2021-02-12T17:57:52.597995+0000","last_fullsized":"2021-02-12T17:57:52.577093+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"2","objects":2856}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":11478,"num_bytes_recovered":47009816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":230141952,"data_stored":230133808,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":112632,"kb_used_data":68784,"kb_used_omap":0,"kb_used_meta":43840,"kb_avail":94259208,"statfs":{"total":96636764160,"available":96521428992,"internally_reserved":0,"allocated":70434816,"data_stored":70309374,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":44892160},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38100000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":132864,"kb_used_data":90168,"kb_used_omap":0,"kb_used_meta":42688,"kb_avail":94238976,"statfs":{"total":96636764160,"available":96500711424,"internally_reserved":0,"allocated":92332032,"data_stored":92210662,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":43712512},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45100000000000001}]}]}]}} 2021-02-12T17:58:30.090 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:30.415 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:30.418 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:30.428 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2021-02-12T17:58:30.407805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12458,"num_bytes_recovered":51023896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":247548,"kb_used_data":163308,"kb_used_omap":0,"kb_used_meta":84224,"kb_avail":188496132,"statfs":{"total":193273528320,"available":193020039168,"internally_reserved":0,"allocated":167227392,"data_stored":166953450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86245376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3450,"num_bytes_recovered":14127128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.929703"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2927","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.485762+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:23.485762+0000","last_peered":"2021-02-12T17:58:23.485762+0000","last_clean":"2021-02-12T17:58:23.485762+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:23.485762+0000","last_undegraded":"2021-02-12T17:58:23.485762+0000","last_fullsized":"2021-02-12T17:58:23.485762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2860","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:23.485349+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:23.485349+0000","last_peered":"2021-02-12T17:58:23.485349+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:23.485349+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8607","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483283+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:23.483283+0000","last_peered":"2021-02-12T17:58:23.483283+0000","last_clean":"2021-02-12T17:58:23.483283+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:23.483283+0000","last_undegraded":"2021-02-12T17:58:23.483283+0000","last_fullsized":"2021-02-12T17:58:23.483283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8274","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.491814+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:23.491814+0000","last_peered":"2021-02-12T17:58:23.491814+0000","last_clean":"2021-02-12T17:58:23.491814+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:23.491814+0000","last_undegraded":"2021-02-12T17:58:23.491814+0000","last_fullsized":"2021-02-12T17:58:23.491814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8371","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483905+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:23.483905+0000","last_peered":"2021-02-12T17:58:23.483905+0000","last_clean":"2021-02-12T17:58:23.483905+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:23.483905+0000","last_undegraded":"2021-02-12T17:58:23.483905+0000","last_fullsized":"2021-02-12T17:58:23.483905+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2836","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483642+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:23.483642+0000","last_peered":"2021-02-12T17:58:23.483642+0000","last_clean":"2021-02-12T17:58:23.483642+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:23.483642+0000","last_undegraded":"2021-02-12T17:58:23.483642+0000","last_fullsized":"2021-02-12T17:58:23.483642+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"5298","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:28.430483+0000","last_change":"2021-02-12T17:58:21.756987+0000","last_active":"2021-02-12T17:58:28.430483+0000","last_peered":"2021-02-12T17:58:28.430483+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:28.430483+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1220,"num_bytes_recovered":4997120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1220}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8604","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-12T17:58:24.494682+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:24.494682+0000","last_peered":"2021-02-12T17:58:24.494682+0000","last_clean":"2021-02-12T17:58:24.494682+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:24.494682+0000","last_undegraded":"2021-02-12T17:58:24.494682+0000","last_fullsized":"2021-02-12T17:58:24.494682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12458,"num_bytes_recovered":51023896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":234156032,"data_stored":234147888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109876,"kb_used_data":72620,"kb_used_omap":0,"kb_used_meta":37248,"kb_avail":94261964,"statfs":{"total":96636764160,"available":96524251136,"internally_reserved":0,"allocated":74362880,"data_stored":74225909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38141952},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.379}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137672,"kb_used_data":90688,"kb_used_omap":0,"kb_used_meta":46976,"kb_avail":94234168,"statfs":{"total":96636764160,"available":96495788032,"internally_reserved":0,"allocated":92864512,"data_stored":92727541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48103424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.34599999999999997}]}]}]}} 2021-02-12T17:58:30.429 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:30.761 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:30.762 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:30.774 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":53,"stamp":"2021-02-12T17:58:30.407805+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12458,"num_bytes_recovered":51023896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":247548,"kb_used_data":163308,"kb_used_omap":0,"kb_used_meta":84224,"kb_avail":188496132,"statfs":{"total":193273528320,"available":193020039168,"internally_reserved":0,"allocated":167227392,"data_stored":166953450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86245376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-3450,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3450,"num_bytes_recovered":14127128,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.929703"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2927","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.485762+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:23.485762+0000","last_peered":"2021-02-12T17:58:23.485762+0000","last_clean":"2021-02-12T17:58:23.485762+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:23.485762+0000","last_undegraded":"2021-02-12T17:58:23.485762+0000","last_fullsized":"2021-02-12T17:58:23.485762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2860","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:23.485349+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:23.485349+0000","last_peered":"2021-02-12T17:58:23.485349+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:23.485349+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8607","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483283+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:23.483283+0000","last_peered":"2021-02-12T17:58:23.483283+0000","last_clean":"2021-02-12T17:58:23.483283+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:23.483283+0000","last_undegraded":"2021-02-12T17:58:23.483283+0000","last_fullsized":"2021-02-12T17:58:23.483283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8274","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.491814+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:23.491814+0000","last_peered":"2021-02-12T17:58:23.491814+0000","last_clean":"2021-02-12T17:58:23.491814+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:23.491814+0000","last_undegraded":"2021-02-12T17:58:23.491814+0000","last_fullsized":"2021-02-12T17:58:23.491814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8371","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483905+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:23.483905+0000","last_peered":"2021-02-12T17:58:23.483905+0000","last_clean":"2021-02-12T17:58:23.483905+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:23.483905+0000","last_undegraded":"2021-02-12T17:58:23.483905+0000","last_fullsized":"2021-02-12T17:58:23.483905+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2836","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483642+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:23.483642+0000","last_peered":"2021-02-12T17:58:23.483642+0000","last_clean":"2021-02-12T17:58:23.483642+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:23.483642+0000","last_undegraded":"2021-02-12T17:58:23.483642+0000","last_fullsized":"2021-02-12T17:58:23.483642+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"5298","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:28.430483+0000","last_change":"2021-02-12T17:58:21.756987+0000","last_active":"2021-02-12T17:58:28.430483+0000","last_peered":"2021-02-12T17:58:28.430483+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:28.430483+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1220,"num_bytes_recovered":4997120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1220}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8604","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-12T17:58:24.494682+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:24.494682+0000","last_peered":"2021-02-12T17:58:24.494682+0000","last_clean":"2021-02-12T17:58:24.494682+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:24.494682+0000","last_undegraded":"2021-02-12T17:58:24.494682+0000","last_fullsized":"2021-02-12T17:58:24.494682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12458,"num_bytes_recovered":51023896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":234156032,"data_stored":234147888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109876,"kb_used_data":72620,"kb_used_omap":0,"kb_used_meta":37248,"kb_avail":94261964,"statfs":{"total":96636764160,"available":96524251136,"internally_reserved":0,"allocated":74362880,"data_stored":74225909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38141952},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.379}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137672,"kb_used_data":90688,"kb_used_omap":0,"kb_used_meta":46976,"kb_avail":94234168,"statfs":{"total":96636764160,"available":96495788032,"internally_reserved":0,"allocated":92864512,"data_stored":92727541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48103424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.34599999999999997}]}]}]}} 2021-02-12T17:58:33.776 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:34.114 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:34.117 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:34.127 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":54,"stamp":"2021-02-12T17:58:32.408188+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12458,"num_bytes_recovered":51023896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":247548,"kb_used_data":163308,"kb_used_omap":0,"kb_used_meta":84224,"kb_avail":188496132,"statfs":{"total":193273528320,"available":193020039168,"internally_reserved":0,"allocated":167227392,"data_stored":166953450,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86245376},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-980,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":980,"num_bytes_recovered":4014080,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.926112"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2927","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.485762+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:23.485762+0000","last_peered":"2021-02-12T17:58:23.485762+0000","last_clean":"2021-02-12T17:58:23.485762+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:23.485762+0000","last_undegraded":"2021-02-12T17:58:23.485762+0000","last_fullsized":"2021-02-12T17:58:23.485762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2860","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:23.485349+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:23.485349+0000","last_peered":"2021-02-12T17:58:23.485349+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:23.485349+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8607","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483283+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:23.483283+0000","last_peered":"2021-02-12T17:58:23.483283+0000","last_clean":"2021-02-12T17:58:23.483283+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:23.483283+0000","last_undegraded":"2021-02-12T17:58:23.483283+0000","last_fullsized":"2021-02-12T17:58:23.483283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8274","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.491814+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:23.491814+0000","last_peered":"2021-02-12T17:58:23.491814+0000","last_clean":"2021-02-12T17:58:23.491814+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:23.491814+0000","last_undegraded":"2021-02-12T17:58:23.491814+0000","last_fullsized":"2021-02-12T17:58:23.491814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8371","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483905+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:23.483905+0000","last_peered":"2021-02-12T17:58:23.483905+0000","last_clean":"2021-02-12T17:58:23.483905+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:23.483905+0000","last_undegraded":"2021-02-12T17:58:23.483905+0000","last_fullsized":"2021-02-12T17:58:23.483905+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2836","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483642+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:23.483642+0000","last_peered":"2021-02-12T17:58:23.483642+0000","last_clean":"2021-02-12T17:58:23.483642+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:23.483642+0000","last_undegraded":"2021-02-12T17:58:23.483642+0000","last_fullsized":"2021-02-12T17:58:23.483642+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"5298","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:28.430483+0000","last_change":"2021-02-12T17:58:21.756987+0000","last_active":"2021-02-12T17:58:28.430483+0000","last_peered":"2021-02-12T17:58:28.430483+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:28.430483+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1611,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1220,"num_bytes_recovered":4997120,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":1220}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8604","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-12T17:58:24.494682+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:24.494682+0000","last_peered":"2021-02-12T17:58:24.494682+0000","last_clean":"2021-02-12T17:58:24.494682+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:24.494682+0000","last_undegraded":"2021-02-12T17:58:24.494682+0000","last_fullsized":"2021-02-12T17:58:24.494682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4445,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":12458,"num_bytes_recovered":51023896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":234156032,"data_stored":234147888,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738385,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":109876,"kb_used_data":72620,"kb_used_omap":0,"kb_used_meta":37248,"kb_avail":94261964,"statfs":{"total":96636764160,"available":96524251136,"internally_reserved":0,"allocated":74362880,"data_stored":74225909,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":38141952},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.28699999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.379}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137672,"kb_used_data":90688,"kb_used_omap":0,"kb_used_meta":46976,"kb_avail":94234168,"statfs":{"total":96636764160,"available":96495788032,"internally_reserved":0,"allocated":92864512,"data_stored":92727541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48103424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.46899999999999997},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.34599999999999997}]}]}]}} 2021-02-12T17:58:34.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:34.475 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:34.477 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:34.488 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":55,"stamp":"2021-02-12T17:58:34.408551+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13786,"num_bytes_recovered":56463384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":250772,"kb_used_data":168452,"kb_used_omap":0,"kb_used_meta":82304,"kb_avail":188492908,"statfs":{"total":193273528320,"available":193016737792,"internally_reserved":0,"allocated":172494848,"data_stored":172220906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":84279296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2308,"num_bytes_recovered":9453568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001745"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2927","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.485762+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:23.485762+0000","last_peered":"2021-02-12T17:58:23.485762+0000","last_clean":"2021-02-12T17:58:23.485762+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:23.485762+0000","last_undegraded":"2021-02-12T17:58:23.485762+0000","last_fullsized":"2021-02-12T17:58:23.485762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2860","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:23.485349+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:23.485349+0000","last_peered":"2021-02-12T17:58:23.485349+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:23.485349+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8607","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483283+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:23.483283+0000","last_peered":"2021-02-12T17:58:23.483283+0000","last_clean":"2021-02-12T17:58:23.483283+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:23.483283+0000","last_undegraded":"2021-02-12T17:58:23.483283+0000","last_fullsized":"2021-02-12T17:58:23.483283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8274","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.491814+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:23.491814+0000","last_peered":"2021-02-12T17:58:23.491814+0000","last_clean":"2021-02-12T17:58:23.491814+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:23.491814+0000","last_undegraded":"2021-02-12T17:58:23.491814+0000","last_fullsized":"2021-02-12T17:58:23.491814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8371","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483905+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:23.483905+0000","last_peered":"2021-02-12T17:58:23.483905+0000","last_clean":"2021-02-12T17:58:23.483905+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:23.483905+0000","last_undegraded":"2021-02-12T17:58:23.483905+0000","last_fullsized":"2021-02-12T17:58:23.483905+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2836","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483642+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:23.483642+0000","last_peered":"2021-02-12T17:58:23.483642+0000","last_clean":"2021-02-12T17:58:23.483642+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:23.483642+0000","last_undegraded":"2021-02-12T17:58:23.483642+0000","last_fullsized":"2021-02-12T17:58:23.483642+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"7954","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:33.431540+0000","last_change":"2021-02-12T17:58:21.756987+0000","last_active":"2021-02-12T17:58:33.431540+0000","last_peered":"2021-02-12T17:58:33.431540+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:33.431540+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2548,"num_bytes_recovered":10436608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8604","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-12T17:58:24.494682+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:24.494682+0000","last_peered":"2021-02-12T17:58:24.494682+0000","last_clean":"2021-02-12T17:58:24.494682+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:24.494682+0000","last_undegraded":"2021-02-12T17:58:24.494682+0000","last_fullsized":"2021-02-12T17:58:24.494682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13786,"num_bytes_recovered":56463384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":239591424,"data_stored":239583280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113100,"kb_used_data":77764,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94258740,"statfs":{"total":96636764160,"available":96520949760,"internally_reserved":0,"allocated":79630336,"data_stored":79493365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.38700000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137672,"kb_used_data":90688,"kb_used_omap":0,"kb_used_meta":46976,"kb_avail":94234168,"statfs":{"total":96636764160,"available":96495788032,"internally_reserved":0,"allocated":92864512,"data_stored":92727541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48103424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.307},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.40500000000000003}]}]}]}} 2021-02-12T17:58:37.489 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:37.838 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:37.840 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:37.851 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2021-02-12T17:58:36.408852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13786,"num_bytes_recovered":56463384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":250772,"kb_used_data":168452,"kb_used_omap":0,"kb_used_meta":82304,"kb_avail":188492908,"statfs":{"total":193273528320,"available":193016737792,"internally_reserved":0,"allocated":172494848,"data_stored":172220906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":84279296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2308,"num_bytes_recovered":9453568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001364"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2927","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.485762+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:23.485762+0000","last_peered":"2021-02-12T17:58:23.485762+0000","last_clean":"2021-02-12T17:58:23.485762+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:23.485762+0000","last_undegraded":"2021-02-12T17:58:23.485762+0000","last_fullsized":"2021-02-12T17:58:23.485762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2860","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:23.485349+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:23.485349+0000","last_peered":"2021-02-12T17:58:23.485349+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:23.485349+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8607","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483283+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:23.483283+0000","last_peered":"2021-02-12T17:58:23.483283+0000","last_clean":"2021-02-12T17:58:23.483283+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:23.483283+0000","last_undegraded":"2021-02-12T17:58:23.483283+0000","last_fullsized":"2021-02-12T17:58:23.483283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8274","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.491814+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:23.491814+0000","last_peered":"2021-02-12T17:58:23.491814+0000","last_clean":"2021-02-12T17:58:23.491814+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:23.491814+0000","last_undegraded":"2021-02-12T17:58:23.491814+0000","last_fullsized":"2021-02-12T17:58:23.491814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8371","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483905+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:23.483905+0000","last_peered":"2021-02-12T17:58:23.483905+0000","last_clean":"2021-02-12T17:58:23.483905+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:23.483905+0000","last_undegraded":"2021-02-12T17:58:23.483905+0000","last_fullsized":"2021-02-12T17:58:23.483905+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2836","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483642+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:23.483642+0000","last_peered":"2021-02-12T17:58:23.483642+0000","last_clean":"2021-02-12T17:58:23.483642+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:23.483642+0000","last_undegraded":"2021-02-12T17:58:23.483642+0000","last_fullsized":"2021-02-12T17:58:23.483642+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"7954","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:33.431540+0000","last_change":"2021-02-12T17:58:21.756987+0000","last_active":"2021-02-12T17:58:33.431540+0000","last_peered":"2021-02-12T17:58:33.431540+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:33.431540+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2548,"num_bytes_recovered":10436608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8604","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-12T17:58:24.494682+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:24.494682+0000","last_peered":"2021-02-12T17:58:24.494682+0000","last_clean":"2021-02-12T17:58:24.494682+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:24.494682+0000","last_undegraded":"2021-02-12T17:58:24.494682+0000","last_fullsized":"2021-02-12T17:58:24.494682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13786,"num_bytes_recovered":56463384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":239591424,"data_stored":239583280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113100,"kb_used_data":77764,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94258740,"statfs":{"total":96636764160,"available":96520949760,"internally_reserved":0,"allocated":79630336,"data_stored":79493365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.38700000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137672,"kb_used_data":90688,"kb_used_omap":0,"kb_used_meta":46976,"kb_avail":94234168,"statfs":{"total":96636764160,"available":96495788032,"internally_reserved":0,"allocated":92864512,"data_stored":92727541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48103424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.307},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.40500000000000003}]}]}]}} 2021-02-12T17:58:37.852 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:38.214 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:38.216 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:38.227 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":58,"stamp":"2021-02-12T17:58:36.408852+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13786,"num_bytes_recovered":56463384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":250772,"kb_used_data":168452,"kb_used_omap":0,"kb_used_meta":82304,"kb_avail":188492908,"statfs":{"total":193273528320,"available":193016737792,"internally_reserved":0,"allocated":172494848,"data_stored":172220906,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":84279296},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2308,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2308,"num_bytes_recovered":9453568,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001364"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2927","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.485762+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:23.485762+0000","last_peered":"2021-02-12T17:58:23.485762+0000","last_clean":"2021-02-12T17:58:23.485762+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:23.485762+0000","last_undegraded":"2021-02-12T17:58:23.485762+0000","last_fullsized":"2021-02-12T17:58:23.485762+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"2860","reported_epoch":"20","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:23.485349+0000","last_change":"2021-02-12T17:57:52.729130+0000","last_active":"2021-02-12T17:58:23.485349+0000","last_peered":"2021-02-12T17:58:23.485349+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:23.485349+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2834,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2834}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8607","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483283+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:23.483283+0000","last_peered":"2021-02-12T17:58:23.483283+0000","last_clean":"2021-02-12T17:58:23.483283+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:23.483283+0000","last_undegraded":"2021-02-12T17:58:23.483283+0000","last_fullsized":"2021-02-12T17:58:23.483283+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8274","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.491814+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:23.491814+0000","last_peered":"2021-02-12T17:58:23.491814+0000","last_clean":"2021-02-12T17:58:23.491814+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:23.491814+0000","last_undegraded":"2021-02-12T17:58:23.491814+0000","last_fullsized":"2021-02-12T17:58:23.491814+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8371","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483905+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:23.483905+0000","last_peered":"2021-02-12T17:58:23.483905+0000","last_clean":"2021-02-12T17:58:23.483905+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:23.483905+0000","last_undegraded":"2021-02-12T17:58:23.483905+0000","last_fullsized":"2021-02-12T17:58:23.483905+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2836","reported_epoch":"20","state":"active+clean","last_fresh":"2021-02-12T17:58:23.483642+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:23.483642+0000","last_peered":"2021-02-12T17:58:23.483642+0000","last_clean":"2021-02-12T17:58:23.483642+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:23.483642+0000","last_undegraded":"2021-02-12T17:58:23.483642+0000","last_fullsized":"2021-02-12T17:58:23.483642+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"7954","reported_epoch":"21","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:33.431540+0000","last_change":"2021-02-12T17:58:21.756987+0000","last_active":"2021-02-12T17:58:33.431540+0000","last_peered":"2021-02-12T17:58:33.431540+0000","last_clean":"2021-02-12T17:57:49.032320+0000","last_became_active":"2021-02-12T17:57:52.725512+0000","last_became_peered":"2021-02-12T17:57:52.725512+0000","last_unstale":"2021-02-12T17:58:33.431540+0000","last_undegraded":"2021-02-12T17:57:52.608040+0000","last_fullsized":"2021-02-12T17:57:52.578067+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":283,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2548,"num_bytes_recovered":10436608,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2548}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8604","reported_epoch":"21","state":"active+clean","last_fresh":"2021-02-12T17:58:24.494682+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:24.494682+0000","last_peered":"2021-02-12T17:58:24.494682+0000","last_clean":"2021-02-12T17:58:24.494682+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:24.494682+0000","last_undegraded":"2021-02-12T17:58:24.494682+0000","last_fullsized":"2021-02-12T17:58:24.494682+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3117,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":13786,"num_bytes_recovered":56463384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":239591424,"data_stored":239583280,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738386,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":113100,"kb_used_data":77764,"kb_used_omap":0,"kb_used_meta":35328,"kb_avail":94258740,"statfs":{"total":96636764160,"available":96520949760,"internally_reserved":0,"allocated":79630336,"data_stored":79493365,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":36175872},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.35899999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.38700000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738390,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137672,"kb_used_data":90688,"kb_used_omap":0,"kb_used_meta":46976,"kb_avail":94234168,"statfs":{"total":96636764160,"available":96495788032,"internally_reserved":0,"allocated":92864512,"data_stored":92727541,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48103424},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.307},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.40500000000000003}]}]}]}} 2021-02-12T17:58:41.229 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:41.574 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:41.575 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:41.588 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2021-02-12T17:58:40.409487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14951,"num_bytes_recovered":61235224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":266040,"kb_used_data":173416,"kb_used_omap":0,"kb_used_meta":92608,"kb_avail":188477640,"statfs":{"total":193273528320,"available":193001103360,"internally_reserved":0,"allocated":177577984,"data_stored":177280868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94830592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":10211328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001299"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2929","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567627+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:34.567627+0000","last_peered":"2021-02-12T17:58:34.567627+0000","last_clean":"2021-02-12T17:58:34.567627+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:34.567627+0000","last_undegraded":"2021-02-12T17:58:34.567627+0000","last_fullsized":"2021-02-12T17:58:34.567627+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"4627","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:38.430773+0000","last_change":"2021-02-12T17:58:34.171822+0000","last_active":"2021-02-12T17:58:38.430773+0000","last_peered":"2021-02-12T17:58:38.430773+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:38.430773+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":882}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8609","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567166+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:34.567166+0000","last_peered":"2021-02-12T17:58:34.567166+0000","last_clean":"2021-02-12T17:58:34.567166+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:34.567166+0000","last_undegraded":"2021-02-12T17:58:34.567166+0000","last_fullsized":"2021-02-12T17:58:34.567166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8276","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567561+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:34.567561+0000","last_peered":"2021-02-12T17:58:34.567561+0000","last_clean":"2021-02-12T17:58:34.567561+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:34.567561+0000","last_undegraded":"2021-02-12T17:58:34.567561+0000","last_fullsized":"2021-02-12T17:58:34.567561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8373","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567603+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:34.567603+0000","last_peered":"2021-02-12T17:58:34.567603+0000","last_clean":"2021-02-12T17:58:34.567603+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:34.567603+0000","last_undegraded":"2021-02-12T17:58:34.567603+0000","last_fullsized":"2021-02-12T17:58:34.567603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2838","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567689+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:34.567689+0000","last_peered":"2021-02-12T17:58:34.567689+0000","last_clean":"2021-02-12T17:58:34.567689+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:34.567689+0000","last_undegraded":"2021-02-12T17:58:34.567689+0000","last_fullsized":"2021-02-12T17:58:34.567689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"8531","reported_epoch":"23","state":"active+clean","last_fresh":"2021-02-12T17:58:35.580125+0000","last_change":"2021-02-12T17:58:35.580125+0000","last_active":"2021-02-12T17:58:35.580125+0000","last_peered":"2021-02-12T17:58:35.580125+0000","last_clean":"2021-02-12T17:58:35.580125+0000","last_became_active":"2021-02-12T17:58:35.577775+0000","last_became_peered":"2021-02-12T17:58:35.577775+0000","last_unstale":"2021-02-12T17:58:35.580125+0000","last_undegraded":"2021-02-12T17:58:35.580125+0000","last_fullsized":"2021-02-12T17:58:35.580125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":11595776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8606","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567196+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:34.567196+0000","last_peered":"2021-02-12T17:58:34.567196+0000","last_clean":"2021-02-12T17:58:34.567196+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:34.567196+0000","last_undegraded":"2021-02-12T17:58:34.567196+0000","last_fullsized":"2021-02-12T17:58:34.567196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14951,"num_bytes_recovered":61235224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":244367360,"data_stored":244359216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128288,"kb_used_data":82712,"kb_used_omap":0,"kb_used_meta":45568,"kb_avail":94243552,"statfs":{"total":96636764160,"available":96505397248,"internally_reserved":0,"allocated":84697088,"data_stored":84548530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46661632},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.51800000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137752,"kb_used_data":90704,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":94234088,"statfs":{"total":96636764160,"available":96495706112,"internally_reserved":0,"allocated":92880896,"data_stored":92732338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.436}]}]}]}} 2021-02-12T17:58:41.589 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:41.922 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:41.924 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:41.936 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":60,"stamp":"2021-02-12T17:58:40.409487+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14951,"num_bytes_recovered":61235224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":266040,"kb_used_data":173416,"kb_used_omap":0,"kb_used_meta":92608,"kb_avail":188477640,"statfs":{"total":193273528320,"available":193001103360,"internally_reserved":0,"allocated":177577984,"data_stored":177280868,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94830592},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2493,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2493,"num_bytes_recovered":10211328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001299"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2929","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567627+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:34.567627+0000","last_peered":"2021-02-12T17:58:34.567627+0000","last_clean":"2021-02-12T17:58:34.567627+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:34.567627+0000","last_undegraded":"2021-02-12T17:58:34.567627+0000","last_fullsized":"2021-02-12T17:58:34.567627+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"4627","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:38.430773+0000","last_change":"2021-02-12T17:58:34.171822+0000","last_active":"2021-02-12T17:58:38.430773+0000","last_peered":"2021-02-12T17:58:38.430773+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:38.430773+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":882,"num_bytes_recovered":3612672,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":882}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8609","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567166+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:34.567166+0000","last_peered":"2021-02-12T17:58:34.567166+0000","last_clean":"2021-02-12T17:58:34.567166+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:34.567166+0000","last_undegraded":"2021-02-12T17:58:34.567166+0000","last_fullsized":"2021-02-12T17:58:34.567166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8276","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567561+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:34.567561+0000","last_peered":"2021-02-12T17:58:34.567561+0000","last_clean":"2021-02-12T17:58:34.567561+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:34.567561+0000","last_undegraded":"2021-02-12T17:58:34.567561+0000","last_fullsized":"2021-02-12T17:58:34.567561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8373","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567603+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:34.567603+0000","last_peered":"2021-02-12T17:58:34.567603+0000","last_clean":"2021-02-12T17:58:34.567603+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:34.567603+0000","last_undegraded":"2021-02-12T17:58:34.567603+0000","last_fullsized":"2021-02-12T17:58:34.567603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2838","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567689+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:34.567689+0000","last_peered":"2021-02-12T17:58:34.567689+0000","last_clean":"2021-02-12T17:58:34.567689+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:34.567689+0000","last_undegraded":"2021-02-12T17:58:34.567689+0000","last_fullsized":"2021-02-12T17:58:34.567689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"8531","reported_epoch":"23","state":"active+clean","last_fresh":"2021-02-12T17:58:35.580125+0000","last_change":"2021-02-12T17:58:35.580125+0000","last_active":"2021-02-12T17:58:35.580125+0000","last_peered":"2021-02-12T17:58:35.580125+0000","last_clean":"2021-02-12T17:58:35.580125+0000","last_became_active":"2021-02-12T17:58:35.577775+0000","last_became_peered":"2021-02-12T17:58:35.577775+0000","last_unstale":"2021-02-12T17:58:35.580125+0000","last_undegraded":"2021-02-12T17:58:35.580125+0000","last_fullsized":"2021-02-12T17:58:35.580125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":11595776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8606","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567196+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:34.567196+0000","last_peered":"2021-02-12T17:58:34.567196+0000","last_clean":"2021-02-12T17:58:34.567196+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:34.567196+0000","last_undegraded":"2021-02-12T17:58:34.567196+0000","last_fullsized":"2021-02-12T17:58:34.567196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1952,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":14951,"num_bytes_recovered":61235224,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":244367360,"data_stored":244359216,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738387,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":128288,"kb_used_data":82712,"kb_used_omap":0,"kb_used_meta":45568,"kb_avail":94243552,"statfs":{"total":96636764160,"available":96505397248,"internally_reserved":0,"allocated":84697088,"data_stored":84548530,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":46661632},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.48999999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.51800000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738391,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137752,"kb_used_data":90704,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":94234088,"statfs":{"total":96636764160,"available":96495706112,"internally_reserved":0,"allocated":92880896,"data_stored":92732338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.436}]}]}]}} 2021-02-12T17:58:44.938 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:45.294 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:45.296 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:45.307 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2021-02-12T17:58:44.410073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16107,"num_bytes_recovered":65970200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":264244,"kb_used_data":177956,"kb_used_omap":0,"kb_used_meta":86272,"kb_avail":188479436,"statfs":{"total":193273528320,"available":193002942464,"internally_reserved":0,"allocated":182226944,"data_stored":181929828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88342528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":9506816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.844684"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2929","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567627+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:34.567627+0000","last_peered":"2021-02-12T17:58:34.567627+0000","last_clean":"2021-02-12T17:58:34.567627+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:34.567627+0000","last_undegraded":"2021-02-12T17:58:34.567627+0000","last_fullsized":"2021-02-12T17:58:34.567627+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"6939","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:43.433040+0000","last_change":"2021-02-12T17:58:34.171822+0000","last_active":"2021-02-12T17:58:43.433040+0000","last_peered":"2021-02-12T17:58:43.433040+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:43.433040+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":8347648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8609","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567166+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:34.567166+0000","last_peered":"2021-02-12T17:58:34.567166+0000","last_clean":"2021-02-12T17:58:34.567166+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:34.567166+0000","last_undegraded":"2021-02-12T17:58:34.567166+0000","last_fullsized":"2021-02-12T17:58:34.567166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8276","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567561+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:34.567561+0000","last_peered":"2021-02-12T17:58:34.567561+0000","last_clean":"2021-02-12T17:58:34.567561+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:34.567561+0000","last_undegraded":"2021-02-12T17:58:34.567561+0000","last_fullsized":"2021-02-12T17:58:34.567561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8373","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567603+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:34.567603+0000","last_peered":"2021-02-12T17:58:34.567603+0000","last_clean":"2021-02-12T17:58:34.567603+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:34.567603+0000","last_undegraded":"2021-02-12T17:58:34.567603+0000","last_fullsized":"2021-02-12T17:58:34.567603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2838","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567689+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:34.567689+0000","last_peered":"2021-02-12T17:58:34.567689+0000","last_clean":"2021-02-12T17:58:34.567689+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:34.567689+0000","last_undegraded":"2021-02-12T17:58:34.567689+0000","last_fullsized":"2021-02-12T17:58:34.567689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"8531","reported_epoch":"23","state":"active+clean","last_fresh":"2021-02-12T17:58:35.580125+0000","last_change":"2021-02-12T17:58:35.580125+0000","last_active":"2021-02-12T17:58:35.580125+0000","last_peered":"2021-02-12T17:58:35.580125+0000","last_clean":"2021-02-12T17:58:35.580125+0000","last_became_active":"2021-02-12T17:58:35.577775+0000","last_became_peered":"2021-02-12T17:58:35.577775+0000","last_unstale":"2021-02-12T17:58:35.580125+0000","last_undegraded":"2021-02-12T17:58:35.580125+0000","last_fullsized":"2021-02-12T17:58:35.580125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":11595776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8606","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567196+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:34.567196+0000","last_peered":"2021-02-12T17:58:34.567196+0000","last_clean":"2021-02-12T17:58:34.567196+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:34.567196+0000","last_undegraded":"2021-02-12T17:58:34.567196+0000","last_fullsized":"2021-02-12T17:58:34.567196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16107,"num_bytes_recovered":65970200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249098240,"data_stored":249090096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126492,"kb_used_data":87252,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":94245348,"statfs":{"total":96636764160,"available":96507236352,"internally_reserved":0,"allocated":89346048,"data_stored":89197490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42799999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137752,"kb_used_data":90704,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":94234088,"statfs":{"total":96636764160,"available":96495706112,"internally_reserved":0,"allocated":92880896,"data_stored":92732338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.39100000000000001}]}]}]}} 2021-02-12T17:58:45.308 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:45.632 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:45.634 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:45.646 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":62,"stamp":"2021-02-12T17:58:44.410073+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16107,"num_bytes_recovered":65970200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":264244,"kb_used_data":177956,"kb_used_omap":0,"kb_used_meta":86272,"kb_avail":188479436,"statfs":{"total":193273528320,"available":193002942464,"internally_reserved":0,"allocated":182226944,"data_stored":181929828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88342528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2321,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2321,"num_bytes_recovered":9506816,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.844684"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2929","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567627+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:34.567627+0000","last_peered":"2021-02-12T17:58:34.567627+0000","last_clean":"2021-02-12T17:58:34.567627+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:34.567627+0000","last_undegraded":"2021-02-12T17:58:34.567627+0000","last_fullsized":"2021-02-12T17:58:34.567627+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"6939","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:43.433040+0000","last_change":"2021-02-12T17:58:34.171822+0000","last_active":"2021-02-12T17:58:43.433040+0000","last_peered":"2021-02-12T17:58:43.433040+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:43.433040+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":8347648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8609","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567166+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:34.567166+0000","last_peered":"2021-02-12T17:58:34.567166+0000","last_clean":"2021-02-12T17:58:34.567166+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:34.567166+0000","last_undegraded":"2021-02-12T17:58:34.567166+0000","last_fullsized":"2021-02-12T17:58:34.567166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8276","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567561+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:34.567561+0000","last_peered":"2021-02-12T17:58:34.567561+0000","last_clean":"2021-02-12T17:58:34.567561+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:34.567561+0000","last_undegraded":"2021-02-12T17:58:34.567561+0000","last_fullsized":"2021-02-12T17:58:34.567561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8373","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567603+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:34.567603+0000","last_peered":"2021-02-12T17:58:34.567603+0000","last_clean":"2021-02-12T17:58:34.567603+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:34.567603+0000","last_undegraded":"2021-02-12T17:58:34.567603+0000","last_fullsized":"2021-02-12T17:58:34.567603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2838","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567689+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:34.567689+0000","last_peered":"2021-02-12T17:58:34.567689+0000","last_clean":"2021-02-12T17:58:34.567689+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:34.567689+0000","last_undegraded":"2021-02-12T17:58:34.567689+0000","last_fullsized":"2021-02-12T17:58:34.567689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"8531","reported_epoch":"23","state":"active+clean","last_fresh":"2021-02-12T17:58:35.580125+0000","last_change":"2021-02-12T17:58:35.580125+0000","last_active":"2021-02-12T17:58:35.580125+0000","last_peered":"2021-02-12T17:58:35.580125+0000","last_clean":"2021-02-12T17:58:35.580125+0000","last_became_active":"2021-02-12T17:58:35.577775+0000","last_became_peered":"2021-02-12T17:58:35.577775+0000","last_unstale":"2021-02-12T17:58:35.580125+0000","last_undegraded":"2021-02-12T17:58:35.580125+0000","last_fullsized":"2021-02-12T17:58:35.580125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":11595776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8606","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567196+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:34.567196+0000","last_peered":"2021-02-12T17:58:34.567196+0000","last_clean":"2021-02-12T17:58:34.567196+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:34.567196+0000","last_undegraded":"2021-02-12T17:58:34.567196+0000","last_fullsized":"2021-02-12T17:58:34.567196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16107,"num_bytes_recovered":65970200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249098240,"data_stored":249090096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126492,"kb_used_data":87252,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":94245348,"statfs":{"total":96636764160,"available":96507236352,"internally_reserved":0,"allocated":89346048,"data_stored":89197490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42799999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137752,"kb_used_data":90704,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":94234088,"statfs":{"total":96636764160,"available":96495706112,"internally_reserved":0,"allocated":92880896,"data_stored":92732338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.39100000000000001}]}]}]}} 2021-02-12T17:58:48.648 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:48.949 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:48.950 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:48.961 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2021-02-12T17:58:48.410822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16107,"num_bytes_recovered":65970200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":264244,"kb_used_data":177956,"kb_used_omap":0,"kb_used_meta":86272,"kb_avail":188479436,"statfs":{"total":193273528320,"available":193002942464,"internally_reserved":0,"allocated":182226944,"data_stored":181929828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88342528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":4734976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001335"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2929","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567627+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:34.567627+0000","last_peered":"2021-02-12T17:58:34.567627+0000","last_clean":"2021-02-12T17:58:34.567627+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:34.567627+0000","last_undegraded":"2021-02-12T17:58:34.567627+0000","last_fullsized":"2021-02-12T17:58:34.567627+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"6939","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:43.433040+0000","last_change":"2021-02-12T17:58:34.171822+0000","last_active":"2021-02-12T17:58:43.433040+0000","last_peered":"2021-02-12T17:58:43.433040+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:43.433040+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":8347648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8609","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567166+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:34.567166+0000","last_peered":"2021-02-12T17:58:34.567166+0000","last_clean":"2021-02-12T17:58:34.567166+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:34.567166+0000","last_undegraded":"2021-02-12T17:58:34.567166+0000","last_fullsized":"2021-02-12T17:58:34.567166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8276","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567561+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:34.567561+0000","last_peered":"2021-02-12T17:58:34.567561+0000","last_clean":"2021-02-12T17:58:34.567561+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:34.567561+0000","last_undegraded":"2021-02-12T17:58:34.567561+0000","last_fullsized":"2021-02-12T17:58:34.567561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8373","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567603+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:34.567603+0000","last_peered":"2021-02-12T17:58:34.567603+0000","last_clean":"2021-02-12T17:58:34.567603+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:34.567603+0000","last_undegraded":"2021-02-12T17:58:34.567603+0000","last_fullsized":"2021-02-12T17:58:34.567603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2838","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567689+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:34.567689+0000","last_peered":"2021-02-12T17:58:34.567689+0000","last_clean":"2021-02-12T17:58:34.567689+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:34.567689+0000","last_undegraded":"2021-02-12T17:58:34.567689+0000","last_fullsized":"2021-02-12T17:58:34.567689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"8531","reported_epoch":"23","state":"active+clean","last_fresh":"2021-02-12T17:58:35.580125+0000","last_change":"2021-02-12T17:58:35.580125+0000","last_active":"2021-02-12T17:58:35.580125+0000","last_peered":"2021-02-12T17:58:35.580125+0000","last_clean":"2021-02-12T17:58:35.580125+0000","last_became_active":"2021-02-12T17:58:35.577775+0000","last_became_peered":"2021-02-12T17:58:35.577775+0000","last_unstale":"2021-02-12T17:58:35.580125+0000","last_undegraded":"2021-02-12T17:58:35.580125+0000","last_fullsized":"2021-02-12T17:58:35.580125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":11595776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8606","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567196+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:34.567196+0000","last_peered":"2021-02-12T17:58:34.567196+0000","last_clean":"2021-02-12T17:58:34.567196+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:34.567196+0000","last_undegraded":"2021-02-12T17:58:34.567196+0000","last_fullsized":"2021-02-12T17:58:34.567196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16107,"num_bytes_recovered":65970200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249098240,"data_stored":249090096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126492,"kb_used_data":87252,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":94245348,"statfs":{"total":96636764160,"available":96507236352,"internally_reserved":0,"allocated":89346048,"data_stored":89197490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42799999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137752,"kb_used_data":90704,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":94234088,"statfs":{"total":96636764160,"available":96495706112,"internally_reserved":0,"allocated":92880896,"data_stored":92732338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.39100000000000001}]}]}]}} 2021-02-12T17:58:48.962 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:49.250 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:49.252 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:49.262 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":66,"stamp":"2021-02-12T17:58:48.410822+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16107,"num_bytes_recovered":65970200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":264244,"kb_used_data":177956,"kb_used_omap":0,"kb_used_meta":86272,"kb_avail":188479436,"statfs":{"total":193273528320,"available":193002942464,"internally_reserved":0,"allocated":182226944,"data_stored":181929828,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88342528},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1156,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1156,"num_bytes_recovered":4734976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001335"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2929","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567627+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:34.567627+0000","last_peered":"2021-02-12T17:58:34.567627+0000","last_clean":"2021-02-12T17:58:34.567627+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:34.567627+0000","last_undegraded":"2021-02-12T17:58:34.567627+0000","last_fullsized":"2021-02-12T17:58:34.567627+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"6939","reported_epoch":"23","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:58:43.433040+0000","last_change":"2021-02-12T17:58:34.171822+0000","last_active":"2021-02-12T17:58:43.433040+0000","last_peered":"2021-02-12T17:58:43.433040+0000","last_clean":"2021-02-12T17:57:49.025648+0000","last_became_active":"2021-02-12T17:57:52.724845+0000","last_became_peered":"2021-02-12T17:57:52.724845+0000","last_unstale":"2021-02-12T17:58:43.433040+0000","last_undegraded":"2021-02-12T17:57:52.606302+0000","last_fullsized":"2021-02-12T17:57:52.578132+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2038,"num_bytes_recovered":8347648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1],"avail_no_missing":["1"],"object_location_counts":[{"shards":"1","objects":2038}],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8609","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567166+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:34.567166+0000","last_peered":"2021-02-12T17:58:34.567166+0000","last_clean":"2021-02-12T17:58:34.567166+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:34.567166+0000","last_undegraded":"2021-02-12T17:58:34.567166+0000","last_fullsized":"2021-02-12T17:58:34.567166+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8276","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567561+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:34.567561+0000","last_peered":"2021-02-12T17:58:34.567561+0000","last_clean":"2021-02-12T17:58:34.567561+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:34.567561+0000","last_undegraded":"2021-02-12T17:58:34.567561+0000","last_fullsized":"2021-02-12T17:58:34.567561+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8373","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567603+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:34.567603+0000","last_peered":"2021-02-12T17:58:34.567603+0000","last_clean":"2021-02-12T17:58:34.567603+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:34.567603+0000","last_undegraded":"2021-02-12T17:58:34.567603+0000","last_fullsized":"2021-02-12T17:58:34.567603+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2838","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567689+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:34.567689+0000","last_peered":"2021-02-12T17:58:34.567689+0000","last_clean":"2021-02-12T17:58:34.567689+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:34.567689+0000","last_undegraded":"2021-02-12T17:58:34.567689+0000","last_fullsized":"2021-02-12T17:58:34.567689+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"8531","reported_epoch":"23","state":"active+clean","last_fresh":"2021-02-12T17:58:35.580125+0000","last_change":"2021-02-12T17:58:35.580125+0000","last_active":"2021-02-12T17:58:35.580125+0000","last_peered":"2021-02-12T17:58:35.580125+0000","last_clean":"2021-02-12T17:58:35.580125+0000","last_became_active":"2021-02-12T17:58:35.577775+0000","last_became_peered":"2021-02-12T17:58:35.577775+0000","last_unstale":"2021-02-12T17:58:35.580125+0000","last_undegraded":"2021-02-12T17:58:35.580125+0000","last_fullsized":"2021-02-12T17:58:35.580125+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":11595776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8606","reported_epoch":"22","state":"active+clean","last_fresh":"2021-02-12T17:58:34.567196+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:34.567196+0000","last_peered":"2021-02-12T17:58:34.567196+0000","last_clean":"2021-02-12T17:58:34.567196+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:34.567196+0000","last_undegraded":"2021-02-12T17:58:34.567196+0000","last_fullsized":"2021-02-12T17:58:34.567196+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16107,"num_bytes_recovered":65970200,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":249098240,"data_stored":249090096,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":126492,"kb_used_data":87252,"kb_used_omap":0,"kb_used_meta":39232,"kb_avail":94245348,"statfs":{"total":96636764160,"available":96507236352,"internally_reserved":0,"allocated":89346048,"data_stored":89197490,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":40173568},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42799999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738392,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137752,"kb_used_data":90704,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":94234088,"statfs":{"total":96636764160,"available":96495706112,"internally_reserved":0,"allocated":92880896,"data_stored":92732338,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.30599999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.39100000000000001}]}]}]}} 2021-02-12T17:58:52.264 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:58:52.566 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:58:52.568 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:58:52.578 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":68,"stamp":"2021-02-12T17:58:52.411579+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16903,"num_bytes_recovered":69230616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":277328,"kb_used_data":181440,"kb_used_omap":0,"kb_used_meta":95872,"kb_avail":188466352,"statfs":{"total":193273528320,"available":192989544448,"internally_reserved":0,"allocated":185794560,"data_stored":185474170,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98172928},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-796,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":796,"num_bytes_recovered":3260416,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001505"},"pg_stats":[{"pgid":"1.7","version":"11'2908","reported_seq":"2931","reported_epoch":"24","state":"active+clean","last_fresh":"2021-02-12T17:58:46.597858+0000","last_change":"2021-02-12T17:57:24.495347+0000","last_active":"2021-02-12T17:58:46.597858+0000","last_peered":"2021-02-12T17:58:46.597858+0000","last_clean":"2021-02-12T17:58:46.597858+0000","last_became_active":"2021-02-12T17:57:24.495047+0000","last_became_peered":"2021-02-12T17:57:24.495047+0000","last_unstale":"2021-02-12T17:58:46.597858+0000","last_undegraded":"2021-02-12T17:58:46.597858+0000","last_fullsized":"2021-02-12T17:58:46.597858+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2908,"ondisk_log_size":2908,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11911168,"num_objects":2908,"num_object_clones":0,"num_object_copies":5816,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2908,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2908,"num_write_kb":11632,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"11'2834","reported_seq":"8542","reported_epoch":"25","state":"active+clean","last_fresh":"2021-02-12T17:58:47.605835+0000","last_change":"2021-02-12T17:58:47.605835+0000","last_active":"2021-02-12T17:58:47.605835+0000","last_peered":"2021-02-12T17:58:47.605835+0000","last_clean":"2021-02-12T17:58:47.605835+0000","last_became_active":"2021-02-12T17:58:47.603228+0000","last_became_peered":"2021-02-12T17:58:47.603228+0000","last_unstale":"2021-02-12T17:58:47.605835+0000","last_undegraded":"2021-02-12T17:58:47.605835+0000","last_fullsized":"2021-02-12T17:58:47.605835+0000","mapping_epoch":24,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2834,"ondisk_log_size":2834,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11608064,"num_objects":2834,"num_object_clones":0,"num_object_copies":5668,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2834,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2834,"num_write_kb":11336,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2834,"num_bytes_recovered":11608064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"11'2857","reported_seq":"8611","reported_epoch":"24","state":"active+clean","last_fresh":"2021-02-12T17:58:46.599917+0000","last_change":"2021-02-12T17:58:10.496930+0000","last_active":"2021-02-12T17:58:46.599917+0000","last_peered":"2021-02-12T17:58:46.599917+0000","last_clean":"2021-02-12T17:58:46.599917+0000","last_became_active":"2021-02-12T17:58:10.494143+0000","last_became_peered":"2021-02-12T17:58:10.494143+0000","last_unstale":"2021-02-12T17:58:46.599917+0000","last_undegraded":"2021-02-12T17:58:46.599917+0000","last_fullsized":"2021-02-12T17:58:46.599917+0000","mapping_epoch":17,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2857,"ondisk_log_size":2857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11702272,"num_objects":2857,"num_object_clones":0,"num_object_copies":5714,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2857,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2857,"num_write_kb":11428,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"11'2746","reported_seq":"8278","reported_epoch":"24","state":"active+clean","last_fresh":"2021-02-12T17:58:46.598072+0000","last_change":"2021-02-12T17:58:23.490042+0000","last_active":"2021-02-12T17:58:46.598072+0000","last_peered":"2021-02-12T17:58:46.598072+0000","last_clean":"2021-02-12T17:58:46.598072+0000","last_became_active":"2021-02-12T17:58:23.487789+0000","last_became_peered":"2021-02-12T17:58:23.487789+0000","last_unstale":"2021-02-12T17:58:46.598072+0000","last_undegraded":"2021-02-12T17:58:46.598072+0000","last_fullsized":"2021-02-12T17:58:46.598072+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2746,"ondisk_log_size":2746,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11247616,"num_objects":2746,"num_object_clones":0,"num_object_copies":5492,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2746,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2746,"num_write_kb":10984,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"11'2779","reported_seq":"8375","reported_epoch":"24","state":"active+clean","last_fresh":"2021-02-12T17:58:46.597820+0000","last_change":"2021-02-12T17:58:08.491831+0000","last_active":"2021-02-12T17:58:46.597820+0000","last_peered":"2021-02-12T17:58:46.597820+0000","last_clean":"2021-02-12T17:58:46.597820+0000","last_became_active":"2021-02-12T17:58:08.489580+0000","last_became_peered":"2021-02-12T17:58:08.489580+0000","last_unstale":"2021-02-12T17:58:46.597820+0000","last_undegraded":"2021-02-12T17:58:46.597820+0000","last_fullsized":"2021-02-12T17:58:46.597820+0000","mapping_epoch":15,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2779,"ondisk_log_size":2779,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11382784,"num_objects":2779,"num_object_clones":0,"num_object_copies":5558,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2779,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2779,"num_write_kb":11116,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"11'2817","reported_seq":"2840","reported_epoch":"24","state":"active+clean","last_fresh":"2021-02-12T17:58:46.598196+0000","last_change":"2021-02-12T17:57:24.492828+0000","last_active":"2021-02-12T17:58:46.598196+0000","last_peered":"2021-02-12T17:58:46.598196+0000","last_clean":"2021-02-12T17:58:46.598196+0000","last_became_active":"2021-02-12T17:57:24.492500+0000","last_became_peered":"2021-02-12T17:57:24.492500+0000","last_unstale":"2021-02-12T17:58:46.598196+0000","last_undegraded":"2021-02-12T17:58:46.598196+0000","last_fullsized":"2021-02-12T17:58:46.598196+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2817,"ondisk_log_size":2817,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11538432,"num_objects":2817,"num_object_clones":0,"num_object_copies":5634,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2817,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2817,"num_write_kb":11268,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"11'2831","reported_seq":"8533","reported_epoch":"24","state":"active+clean","last_fresh":"2021-02-12T17:58:46.597851+0000","last_change":"2021-02-12T17:58:35.580125+0000","last_active":"2021-02-12T17:58:46.597851+0000","last_peered":"2021-02-12T17:58:46.597851+0000","last_clean":"2021-02-12T17:58:46.597851+0000","last_became_active":"2021-02-12T17:58:35.577775+0000","last_became_peered":"2021-02-12T17:58:35.577775+0000","last_unstale":"2021-02-12T17:58:46.597851+0000","last_undegraded":"2021-02-12T17:58:46.597851+0000","last_fullsized":"2021-02-12T17:58:46.597851+0000","mapping_epoch":22,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2831,"ondisk_log_size":2831,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11595776,"num_objects":2831,"num_object_clones":0,"num_object_copies":5662,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2831,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2831,"num_write_kb":11324,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2831,"num_bytes_recovered":11595776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"11'2856","reported_seq":"8608","reported_epoch":"24","state":"active+clean","last_fresh":"2021-02-12T17:58:46.599992+0000","last_change":"2021-02-12T17:58:24.494682+0000","last_active":"2021-02-12T17:58:46.599992+0000","last_peered":"2021-02-12T17:58:46.599992+0000","last_clean":"2021-02-12T17:58:46.599992+0000","last_became_active":"2021-02-12T17:58:24.492155+0000","last_became_peered":"2021-02-12T17:58:24.492155+0000","last_unstale":"2021-02-12T17:58:46.599992+0000","last_undegraded":"2021-02-12T17:58:46.599992+0000","last_fullsized":"2021-02-12T17:58:46.599992+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":2856,"ondisk_log_size":2856,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":11694104,"num_objects":2856,"num_object_clones":0,"num_object_copies":5712,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2856,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2856,"num_write_kb":11421,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":92680216,"num_objects":22628,"num_object_clones":0,"num_object_copies":45256,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22628,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":22628,"num_write_kb":90509,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":16903,"num_bytes_recovered":69230616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":252366848,"data_stored":252358704,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":22628,"ondisk_log_size":22628,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738389,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":139560,"kb_used_data":90720,"kb_used_omap":0,"kb_used_meta":48832,"kb_avail":94232280,"statfs":{"total":96636764160,"available":96493854720,"internally_reserved":0,"allocated":92897280,"data_stored":92737085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":50003968},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42699999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":8,"seq":34359738393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":137768,"kb_used_data":90720,"kb_used_omap":0,"kb_used_meta":47040,"kb_avail":94234072,"statfs":{"total":96636764160,"available":96495689728,"internally_reserved":0,"allocated":92897280,"data_stored":92737085,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":48168960},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.49399999999999999}]},{"osd":2,"last update":"Fri Feb 12 17:58:23 2021","interfaces":[{"interface":"back","average":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"min":{"1min":0.29799999999999999,"5min":0.29799999999999999,"15min":0.29799999999999999},"max":{"1min":3.6509999999999998,"5min":3.6509999999999998,"15min":3.6509999999999998},"last":0.36199999999999999},{"interface":"front","average":{"1min":1.2729999999999999,"5min":1.2729999999999999,"15min":1.2729999999999999},"min":{"1min":0.314,"5min":0.314,"15min":0.314},"max":{"1min":6.6859999999999999,"5min":6.6859999999999999,"15min":6.6859999999999999},"last":0.40600000000000003}]}]}]}} 2021-02-12T17:58:52.579 INFO:tasks.osd_backfill.ceph_manager:recovered! 2021-02-12T17:58:52.579 INFO:tasks.osd_backfill:rados -p rbd bench 30 write -b 4096 --no-cleanup 2021-02-12T17:58:52.579 DEBUG:teuthology.orchestra.run.smithi107:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage rados -p rbd bench 30 write -b 4096 --no-cleanup 2021-02-12T17:58:52.621 INFO:teuthology.orchestra.run.smithi107.stdout:hints = 1 2021-02-12T17:58:52.622 INFO:teuthology.orchestra.run.smithi107.stdout:Maintaining 16 concurrent writes of 4096 bytes to objects of size 4096 for up to 30 seconds or 0 objects 2021-02-12T17:58:52.622 INFO:teuthology.orchestra.run.smithi107.stdout:Object prefix: benchmark_data_smithi107_39540 2021-02-12T17:58:52.622 INFO:teuthology.orchestra.run.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-02-12T17:58:52.622 INFO:teuthology.orchestra.run.smithi107.stdout: 0 0 0 0 0 0 - 0 2021-02-12T17:58:52.694 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:52.692+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x555983093ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:52.715 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:52.712+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8039790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:53.025 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:53.023+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc80413c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:58:53.025 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:53.024+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8039790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:53.030 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:53.028+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:53.095 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:53.093+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:53.622 INFO:teuthology.orchestra.run.smithi107.stdout: 1 16 979 963 3.76151 3.76172 0.0180892 0.0164905 2021-02-12T17:58:53.697 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:53.696+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:53.831 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:53.829+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc8043790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:53.837 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:53.835+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc8043790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:58:53.927 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:53.925+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:54.176 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:54.174+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc8039750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:58:54.178 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:54.176+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:54.209 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:54.207+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8043380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:54.358 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:54.356+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803a270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:58:54.622 INFO:teuthology.orchestra.run.smithi107.stdout: 2 16 2136 2120 4.14035 4.51953 0.0109725 0.0150188 2021-02-12T17:58:54.632 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:54.631+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:54.725 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:54.723+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:58:54.937 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:54.935+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc8043380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:54.976 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:54.974+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:55.007 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.005+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8043380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:55.081 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.079+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:55.291 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.290+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8041130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:55.388 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.387+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:55.623 INFO:teuthology.orchestra.run.smithi107.stdout: 3 16 3175 3159 4.11299 4.05859 0.0144699 0.0151465 2021-02-12T17:58:55.749 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.747+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:58:55.761 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.759+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:55.771 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.770+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:55.783 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.782+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8041130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:55.878 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:55.876+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.089 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.087+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8039220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.114 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.113+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc8042060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:58:56.134 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.133+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8039220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.166 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.164+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc8042020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.288 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.286+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.316 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.315+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8039220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.322 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.321+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.479 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.478+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8039220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.521 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.520+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:56.622 INFO:teuthology.orchestra.run.smithi107.stdout: 4 16 4281 4265 4.16474 4.32031 0.0154072 0.0149805 2021-02-12T17:58:56.934 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:56.933+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:57.049 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:57.047+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:57.062 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:57.060+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:57.445 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:57.444+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:57.503 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:57.501+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:57.622 INFO:teuthology.orchestra.run.smithi107.stdout: 5 16 5362 5346 4.17626 4.22266 0.0112406 0.0149454 2021-02-12T17:58:57.689 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:57.687+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8041e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:57.836 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:57.835+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8041e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:57.846 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:57.846+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:58:58.622 INFO:teuthology.orchestra.run.smithi107.stdout: 6 16 6417 6401 4.16702 4.12109 0.0124594 0.0149708 2021-02-12T17:58:58.835 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:58.835+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc8041e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:59.622 INFO:teuthology.orchestra.run.smithi107.stdout: 7 16 7249 7233 4.03597 3.25 0.0113186 0.0154589 2021-02-12T17:58:59.626 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:59.625+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:59.669 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:59.669+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:58:59.711 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:58:59.710+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:00.084 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:00.083+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803e110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:00.241 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:00.240+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc8041cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:00.261 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:00.261+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:00.431 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:00.431+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc8041cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:00.438 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:00.438+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:00.458 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:00.457+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:00.622 INFO:teuthology.orchestra.run.smithi107.stdout: 8 16 8004 7988 3.9001 2.94922 0.0141903 0.0160035 2021-02-12T17:59:01.046 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:01.045+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:01.089 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:01.089+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:01.496 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:01.495+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:01.622 INFO:teuthology.orchestra.run.smithi107.stdout: 9 16 9026 9010 3.91018 3.99219 0.00916198 0.0159617 2021-02-12T17:59:01.682 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:01.681+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:01.862 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:01.862+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:02.077 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:02.077+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:02.166 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:02.166+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:02.197 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:02.195+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:02.595 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:02.595+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:02.622 INFO:teuthology.orchestra.run.smithi107.stdout: 10 16 10021 10005 3.9078 3.88672 0.0282768 0.0159771 2021-02-12T17:59:02.674 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:02.674+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:02.683 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:02.683+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:02.742 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:02.741+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:02.955 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:02.954+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:03.260 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:03.259+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:03.623 INFO:teuthology.orchestra.run.smithi107.stdout: 11 16 10996 10980 3.89876 3.80859 0.0187344 0.0160171 2021-02-12T17:59:03.689 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:03.688+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:03.727 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:03.727+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:03.808 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:03.808+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:04.229 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:04.229+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:04.248 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:04.248+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:04.550 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:04.549+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8037a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:04.624 INFO:teuthology.orchestra.run.smithi107.stdout: 12 16 11944 11928 3.88207 3.70312 0.0182805 0.0160831 2021-02-12T17:59:05.540 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:05.539+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc80381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:05.553 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:05.552+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:05.624 INFO:teuthology.orchestra.run.smithi107.stdout: 13 16 12873 12857 3.86258 3.62891 0.0217926 0.0161693 2021-02-12T17:59:05.836 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:05.836+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:05.837 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:05.836+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:05.906 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:05.906+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:05.994 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:05.993+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:06.090 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:06.088+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:06.125 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:06.124+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc80381b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:06.168 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:06.167+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:06.578 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:06.577+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:06.624 INFO:teuthology.orchestra.run.smithi107.stdout: 14 16 13764 13748 3.83527 3.48047 0.0285172 0.0162826 2021-02-12T17:59:07.582 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph -- tell osd.1 injectargs --objectstore-blackhole=true 2021-02-12T17:59:07.624 INFO:teuthology.orchestra.run.smithi107.stdout: 15 16 14537 14521 3.78087 3.01953 0.0274728 0.0163664 2021-02-12T17:59:07.760 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.757+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.760 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.758+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.760 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.758+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.760 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.759+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.761 INFO:teuthology.orchestra.run.smithi107.stdout:{} 2021-02-12T17:59:07.761 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.760+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.761 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.760+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.764 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.763+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.764 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.764+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.765 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.764+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.765 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.764+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.767 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.764+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.768 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.766+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.766+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.766+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.768+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.768+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.768+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.769+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.769+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.769+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.771 INFO:teuthology.orchestra.run.smithi107.stderr:objectstore_blackhole = 'true' (not observed, change may require restart) 2021-02-12T17:59:07.773 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.771+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.773 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.772+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.773 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.772+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.773 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.772+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.773 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.773+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.774 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.773+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.774 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.774+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.776 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.775+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.776 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.775+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.776 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.775+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.777 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.776+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.779 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.778+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.780 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.779+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.781 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.780+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.782 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.781+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.784 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.781+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.785 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.782+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.785 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.782+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.785 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.783+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.785 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.783+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.785 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.784+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.786 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.785+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.785+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.785+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.786+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.786+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.787+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.788 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.788+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.788 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.788+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.789 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.788+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.789 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.789+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.789 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.789+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.792 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.790+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.793 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.790+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.793 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.791+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.793 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.791+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.793 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.791+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.793 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.791+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.793 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.792+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.794 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.793+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.794 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.793+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.795 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.794+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.795 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.794+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.796 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.796+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.799 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.797+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.800 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.798+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.800 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.798+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.800 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.798+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.801 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.798+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.804 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.800+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.804 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.800+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.804 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.800+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.805 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.801+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.806 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.801+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.806 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.801+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.806 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.801+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.806 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.802+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.806 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.802+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.806 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.802+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.803+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.804+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.805+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.805+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.806+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.806+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.807+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.808+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.808+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.808+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.808+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.808+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.810 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.809+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.812 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.810+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.812 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.810+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.813 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.811+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.813 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.811+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.813 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.811+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.813 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.812+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.813 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.813+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.814 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.813+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.815+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.816+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.816+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.817+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.818+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.818+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.818+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.820 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.819+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.820 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.819+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.820 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.820+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.821 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.820+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.821 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.820+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.821 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.821+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.823 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.822+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.824 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.822+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.824 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.822+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.824 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.823+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.825 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.823+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.825 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.824+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.825 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.825+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.830 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.827+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.832 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.827+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.828+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.828+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.828+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.828+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.829+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.838 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.831+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.841 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.832+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.841 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.832+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.833+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.834+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.834+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.835+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.846 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.837+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.846 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.837+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.847 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.837+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.847 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.837+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.847 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.838+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.847 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.838+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.847 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.838+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.847 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.842+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.848 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.842+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.848 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.842+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.848 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.843+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.849 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.843+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.849 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.847+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.850 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.847+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.851 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.849+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.851 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.849+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.851 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.849+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.852 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.850+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.852 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.850+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.852 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.850+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.852 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.851+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.852 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.852+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.853 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.852+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.853 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.852+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.855 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.852+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.855 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.854+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.855 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.854+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.856 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.855+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.856 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.856+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.856 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.856+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.857 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.856+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.858 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.857+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.858 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.858+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.859 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.858+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.859 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.858+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.859 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.858+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.860 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.859+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.860 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.859+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.860 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.859+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.861 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.860+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.861 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.860+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.861 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.861+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.861 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.861+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.865 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.864+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.865 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.865+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.866 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.865+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.866 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.865+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.866 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.866+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.866 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.866+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.869 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.866+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.872 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.867+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.872 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.867+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.873 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.868+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.868+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.868+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.869+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.869+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.870+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.875 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.871+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.875 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.871+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.875 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.871+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.875 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.871+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.876 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.873+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.876 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.873+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.876 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.873+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.876 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.874+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.876 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.875+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.876 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.875+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.877 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.876+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.877 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.876+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.878+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.880 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.878+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.880 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.879+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.880 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.879+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.881 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.880+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.881 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.880+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.881 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.881+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.882 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.881+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.883 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.882+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.884 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.883+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.884 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.883+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.884 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.883+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.887 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.884+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.887 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.884+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.891 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.886+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.892 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.887+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.892 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.887+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.892 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.888+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.893 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.889+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.894 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.890+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.894 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.890+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.894 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.892+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.894 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.892+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.895 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.893+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.895 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.893+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.895 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.894+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.895 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.895+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.896 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.895+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.896 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.895+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.897 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.896+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.898 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.897+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.898 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.897+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.898 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.897+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.899 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.898+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.899 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.898+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.899 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.898+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.900 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.899+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.900 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.900+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.901+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.901+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.902 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.901+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.903 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.902+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.903 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.902+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.905 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.905+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.906 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.905+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.907 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.907+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.908 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.907+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.908 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.907+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.909 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.908+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.909 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.908+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.909 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.908+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.910 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.909+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.915 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.910+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.915 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.910+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.917 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.910+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.917 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.911+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.917 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.911+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.917 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.912+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.912+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.912+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.913+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.913+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.913+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.914+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.914+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.914+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.915+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.916+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.916+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.916+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.918 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.917+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.919 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.917+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.919 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.917+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.919 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.918+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.919 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.919+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.922 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.919+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.922 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.920+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.922 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.921+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.924 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.924+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.924 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.924+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.925 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.924+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.925 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.924+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.925 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.925+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.926 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.925+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.926 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.926+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.926 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.926+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.927 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.927+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.928 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.927+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.929+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.929+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.930 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.929+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.930 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.929+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.930 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.930+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.930 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.930+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.933 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.930+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.933 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.930+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.933 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.932+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.934 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.934+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.935 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.934+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.935 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.934+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.935 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.935+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.936+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.937+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.938+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.938+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.938+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.939+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.940+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.944 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.940+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.944 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.940+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.944 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.940+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.944 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.938+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.945 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.941+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.945 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.941+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.945 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.941+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.945 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.942+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.946 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.943+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.946 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.943+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.946 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.943+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.946 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.944+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.946 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.944+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.946 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.944+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.948 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.947+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.948 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.947+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.948 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.947+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.949 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.949+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.949 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.949+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.950 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.949+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.951 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.950+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.951 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.950+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.951 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.950+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.952 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.952+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.953 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.952+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.954 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.953+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.954 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.954+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.955 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.954+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.955 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.954+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.955 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.954+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.955 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.954+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.956 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.955+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.956 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.956+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.956+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.957+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.958 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.957+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.958 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.957+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.959 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.958+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.961 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.960+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.962 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.961+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.962 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.961+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.962 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.961+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.963 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.962+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.966 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.966+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.968 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.967+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.968 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.967+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.968 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.967+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.969 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.968+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.969 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.968+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.969 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.968+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.970 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.969+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.970 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.970+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.970 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.970+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.970 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.970+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.971 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.971+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.972 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.971+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.972 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.972+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.975 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.972+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.976 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.973+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.977 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.973+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.977 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.974+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.977 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.974+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.977 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.975+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.975+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.975+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.979 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.976+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.979 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.976+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.985 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.977+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.985 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.977+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.978+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.978+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.978+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.979+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.980+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.980+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.980+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.980+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.981+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.981+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.994 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.983+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.994 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.985+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.994 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.987+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.994 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.987+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.994 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.987+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.987+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.987+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.988+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.988+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.988+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.988+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.989+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.996 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.990+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.996 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.992+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.993+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.993+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.995+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.995+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.996+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.997+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.997+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.997+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:07.999 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:07.998+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.004 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.000+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.004 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.000+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.005 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.000+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.005 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.001+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.005 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.001+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.005 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.002+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.005 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.002+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.005 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.002+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.003+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.003+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.003+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.005+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.005+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.005+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.005+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.006+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.006+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.006+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.006+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.006+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.008 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.007+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.008 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.008+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.009 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.008+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.010 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.009+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.011 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.011+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.013 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.011+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.013 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.011+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.014 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.012+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.014 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.013+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.014 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.013+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.014 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.013+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.015 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.014+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.015 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.015+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.015 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.015+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.016 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.016+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.020 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.016+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.020 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.017+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.020 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.018+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.023 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.018+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.023 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.019+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.023 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.019+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.024 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.020+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.024 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.020+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.024 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.021+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.024 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.021+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.024 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.021+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.024 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.022+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.025 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.022+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.025 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.024+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.026 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.025+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.027 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.027+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.028 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.027+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.028 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.028+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.028 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.028+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.028 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.028+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.029 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.029+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.030 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.029+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.030 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.030+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.032+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.032+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.033+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.033+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.033+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.034+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.034+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.035+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.038 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.036+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.038 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.037+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.038 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.037+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.038 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.037+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.039 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.039+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.040 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.040+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.041 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.040+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.043 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.041+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.043 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.042+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.044 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.042+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.044 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.043+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.044 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.043+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.044 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.044+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.048 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.045+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.048 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.045+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.049 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.046+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.049 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.046+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.049 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.047+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.049 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.047+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.051 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.049+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.051 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.049+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.052 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.050+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.052 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.050+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.055 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.052+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.055 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.052+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.056 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.053+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.056 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.053+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.056 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.053+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.054+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.055+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.055+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.055+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.055+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.056+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.056+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.057+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.057+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.058+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.058+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.058+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.058+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.062 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.059+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.062 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.060+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.063 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.062+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.063 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.062+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.064 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.063+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.065+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.066+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.066+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.066+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.066+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.068+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.068+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.071 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.069+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.071 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.070+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.072 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.070+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.073 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.072+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.073 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.072+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.073 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.072+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.076 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.074+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.077 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.075+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.077 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.075+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.077 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.075+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.079 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.078+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.079 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.078+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.079 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.078+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.079 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.078+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.080 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.079+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.080 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.079+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.080+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.080+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.080+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.081+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.081+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.084 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.083+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.086 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.085+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.086 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.086+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.087 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.086+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.087 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.086+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.087 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.087+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.088 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.087+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.088 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.087+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.091 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.088+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.091 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.089+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.092 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.089+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.092 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.089+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.092 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.089+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.094 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.090+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.090+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.092+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.092+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.093+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.093+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.094+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.094+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.095+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.096 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.095+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.100 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.097+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.100 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.097+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.101 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.098+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.102 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.100+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.102 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.100+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.102 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.101+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.102 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.102+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.103 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.102+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.103 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.103+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.104 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.103+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.104 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.104+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.105 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.105+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.106 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.105+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.106 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.106+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.107 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.106+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.107 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.106+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.108 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.107+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.110 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.108+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.110 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.109+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.111 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.110+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.113 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.112+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.113 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.113+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.113 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.113+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.114 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.113+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.114 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.113+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.114 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.114+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.115 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.114+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.115 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.114+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.119 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.116+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.119 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.116+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.119 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.116+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.119 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.117+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.117+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.117+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.117+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.117+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.117+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.119+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.119+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.120+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.121+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.123 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.123+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.124 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.123+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.124 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.123+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.124 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.124+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.128 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.125+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.128 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.125+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.128 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.126+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.128 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.126+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.128 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.126+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.128 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.127+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.128 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.127+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.129 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.127+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.129 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.128+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.129 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.128+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.130 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.129+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.130 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.129+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.130 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.129+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.130 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:08.129+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:08.131 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.130+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.131 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.131+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.133 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.132+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.134 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.133+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.135 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.135+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.135 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.135+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.135 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.135+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.136 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.135+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.136 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.135+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.137 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.136+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.137 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.136+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.140 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.137+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.141 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.137+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.141 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.139+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.141 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.139+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.141 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.140+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.142 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.140+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.142 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.141+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.142 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.141+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.143 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.142+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.143 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.143+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.144 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.143+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.147 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.144+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.147 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.145+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.147 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.145+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.147 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.146+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.148 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.146+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.148 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.146+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.148 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.147+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.148 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.147+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.148 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.148+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.148 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.148+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.149 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.148+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.149 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.149+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.150 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.149+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.154 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.151+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.155 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.153+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.155 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.153+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.158 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.153+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.158 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.154+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.158 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.154+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.154+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.154+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.155+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.155+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.156+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.160 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.156+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.160 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.156+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.160 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.157+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.160 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.157+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.161 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.158+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.161 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.159+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.161 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.159+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.161 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.160+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.161 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.161+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.162 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.161+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.162 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.161+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.162 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.162+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.163 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.163+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.163 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.163+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.164 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.164+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.165 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.164+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.165 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.165+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.169 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.167+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.170 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.168+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.170 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.169+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.170 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.170+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.173 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.171+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.173 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.171+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.173 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.173+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.173 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.173+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.173 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.173+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.173 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.173+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.174 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.174+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.175 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.175+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.176 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.175+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.176 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.176+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.177 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.177+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.178 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.178+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.178 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.178+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.179 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.178+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.180 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.179+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.180 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.179+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.181 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.181+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.181 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.181+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.183 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.182+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.183 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.182+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.184 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.184+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.184 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.184+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.185 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.184+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.185 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.184+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.185 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.184+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.185 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.184+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.185 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.185+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.186 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.185+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.186 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.186+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.186 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.186+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.187 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.186+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.187 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.186+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.189 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.188+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.190 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.189+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.191 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.190+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.191 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.190+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.191 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.191+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.193 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.192+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.194 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.194+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.194 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.194+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.195 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.195+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.195 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.195+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.197 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.196+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.198 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.197+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.198 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.197+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.201 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.198+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.202 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.198+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.202 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.198+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.202 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.199+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.202 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.200+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.202 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.200+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.202 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.200+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.204 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.201+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.205 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.201+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.205 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.202+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.206 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.202+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.206 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.204+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.206 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.204+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.206 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.204+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.206 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.205+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.206 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.206+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.207 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.207+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.208 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.207+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.208 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.208+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.208 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.208+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.208+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.209+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.210+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.211+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.211+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.211+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.213+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.213+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.214+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.214+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.215+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.215+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.215+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.216+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.217+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.218+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.218+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.220+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.221+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.222+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.222+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.223+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.224+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.224+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.224+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.224+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.225+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.225+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.225+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.225+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.226+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.227+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.228+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.228+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.228+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.229+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.229+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.229+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.230+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.230+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.232+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.233+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.233+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.233+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.234+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.234+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.234+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.235+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.236+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.236+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.237+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.238+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.238+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.239+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.240+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.240+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.242+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.244+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.244+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.244+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.244+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.244+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.244+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.245+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.245+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.246+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.246+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.246+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.247+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.247+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.248+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.248+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.248+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.249+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.251+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.252+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.254+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.254+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.254+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.255+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.256+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.257+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.257+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.258+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.258+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.258+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.259+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.259+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.260+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.261+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.261+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.262+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.262+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.263+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.263+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.264+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.264+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.264+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.267+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.267+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.267+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.267+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.267+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.268+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.269+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.269+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.269+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.270+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.271+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.271+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.272+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.272+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.272+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.273+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.273+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.273+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.273+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.274+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.274+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.277+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.278+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.278+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.279+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.279+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.280+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.280+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.281+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.281+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.281+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.282+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.282+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.283+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.283+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.283+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.284+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.285+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.285+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.285+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.286+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.286+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.286+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.289 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:08.289+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:08.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.287+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.288+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.288+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.289+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.289+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.291+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.291+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.291+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.293+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.293+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.293+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.294+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.295+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.296+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.296+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.297+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.297+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.297+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.298+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.298+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.299+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.300+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.300+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.300+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.300+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.302+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.304+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.304+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.304+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.305+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.306+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.307+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.307+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.307+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.307+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.307+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.308+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.309+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.313 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.312+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.313 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.313+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.313 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.313+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.314 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.314+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.317 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.315+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.317 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.315+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.319 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.318+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.319 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.318+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.321 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.320+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.322 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.321+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.322 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.321+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.322 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.322+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.322 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.322+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.323 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.323+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.324 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.323+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.327 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.323+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.327 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.324+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.327 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.324+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.328 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.324+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.329 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.325+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.329 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.325+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.329 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.325+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.329 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.326+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.330 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.326+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.330 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.326+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.330 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.327+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.330 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.327+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.330 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.327+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.330 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.329+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.331 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.329+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.331 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.329+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.331 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.330+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.332 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.331+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.332 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.331+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.334 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.333+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.334 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.333+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.335 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.335+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.336 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.336+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.337 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.336+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.337 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.337+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.338 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.338+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.338 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.338+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.339 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.339+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.339 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.339+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.345 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.339+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.345 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.340+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.347 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.341+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.348 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.342+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.348 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.342+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.348 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.343+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.348 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.344+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.348 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.344+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.349 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.344+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.349 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.344+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.351 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.344+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.351 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.346+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.351 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.346+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.351 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.346+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.352 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.347+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.352 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.347+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.352 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.348+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.353 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.349+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.353 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.349+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.353 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.349+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.353 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.350+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.353 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.350+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.353 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.351+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.353 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.351+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.354 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.351+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.354 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.351+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.354 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.351+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.354 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.352+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.354 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.352+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.356 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.353+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.356 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.354+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.357 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.354+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.357 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.355+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.357 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.355+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.357 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.355+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.357 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.355+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.357 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.355+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.357 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.356+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.357 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.357+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.359 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.359+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.362 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.359+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.362 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.360+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.362 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.360+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.362 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.362+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.362 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.362+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.363 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.362+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.364 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.363+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.364 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.363+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.366 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.365+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.367 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.366+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.367 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.367+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.367 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.367+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.369 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.367+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.369 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.368+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.370 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.368+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.370 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.368+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.370 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.369+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.370 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.369+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.370 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.370+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.371 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.371+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.371 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.371+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.372 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.372+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.374 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.374+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.374 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.374+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.382 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.375+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.383 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.375+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.415 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.376+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.378+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.378+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.378+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.379+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.379+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.379+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.379+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.380+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.416 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.380+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.417 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.380+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.419 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.382+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.419 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.382+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.419 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.382+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.420 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.383+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.420 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.383+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.420 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.384+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.420 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.384+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.421 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.416+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.421 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.416+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.421 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.416+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.421 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.416+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.424 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.416+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.424 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.418+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.424 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.418+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.425 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.418+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.425 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.420+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.425 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.420+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.425 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.421+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.428 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.421+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.429 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.422+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.429 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.423+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.429 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.423+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.429 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.423+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.430 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.424+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.430 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.425+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.430 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.425+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.430 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.426+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.430 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.427+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.431 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.427+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.431 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.427+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.431 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.427+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.431 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.429+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.431 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.429+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.432 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.429+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.432 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.430+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.434 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.430+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.434 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.431+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.434 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.431+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.434 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.432+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.435 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.433+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.435 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.433+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.435 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.433+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.435 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.434+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.436 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.434+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.436 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.434+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.436 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.435+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.436 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.436+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.437 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.436+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.437 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.436+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.437 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.436+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.438 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.437+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.438 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.438+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.441 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.439+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.441 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.439+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.441 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.439+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.442 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.439+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.442 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.440+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.442 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.441+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.442 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.441+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.443 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.443+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.447 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.444+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.447 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.444+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.447 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.445+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.447 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.445+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.448 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.446+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.448 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.446+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.448 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.446+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.448 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.446+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.448 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.446+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.448 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.448+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.449 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.448+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.450 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.450+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.450 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.450+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.451 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.450+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.451 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.451+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.452 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.451+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.452 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.452+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.453 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.452+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.453 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.452+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.453 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.453+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.454 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.453+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.454 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.453+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.456 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.455+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.456 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.455+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.456 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.455+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.456 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.455+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.457 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.456+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.457 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.456+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.460 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.456+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.460 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.457+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.460 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.458+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.460 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.460+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.461 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.460+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.464 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.461+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.464 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.461+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.464 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.462+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.464 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.463+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.464 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.463+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.464 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.463+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.464 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.463+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.465 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.464+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.465 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.464+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.465 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.465+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.469 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.468+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.469 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.468+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.471 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.469+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.472 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.472+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.472 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.472+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.472 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.472+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.473 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.472+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.473 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.472+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.474 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.473+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.474 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.474+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.474 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.474+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.475 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.474+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.475 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.474+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.475 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.475+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.480 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.475+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.475+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.477+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.477+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.478+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.479+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.479+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.479+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.480+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.480+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.481 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.481+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.483 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.483+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.484 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.483+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.484 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.483+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.485 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.484+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.485 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.484+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.485 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.485+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.486 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.485+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.486 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.486+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.486 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.486+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.487 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.487+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.487 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.487+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.488 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.487+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.488 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.488+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.489 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.488+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.489 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.489+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.490 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.490+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.491 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.490+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.491 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.491+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.494 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.492+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.494 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.492+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.494 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.492+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.494 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.493+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.494 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.493+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.496 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.494+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.496 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.494+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.496 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.495+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.497 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.495+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.497 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.495+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.497 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.496+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.497 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.496+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.497 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.497+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.497 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.497+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.498 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.498+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.499 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.498+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.501 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.499+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.502 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.499+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.502 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.500+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.502 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.500+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.502 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.501+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.502 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.502+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.502 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.502+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.504 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.504+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.506 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.505+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.506 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.505+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.506 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.506+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.507 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.507+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.508 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.507+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.508 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.507+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.512 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.509+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.512 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.509+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.512 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.510+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.512 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.510+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.512 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.510+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.512 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.511+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.513 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.513+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.514 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.513+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.514 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.513+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.514 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.513+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.516 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.516+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.516 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.516+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.518 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.516+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.518 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.516+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.518 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.517+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.518 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.517+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.518 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.518+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.519 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.518+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.519 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.518+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.519 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.518+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.519 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.519+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.531 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.521+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.531 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.522+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.531 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.522+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.531 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.523+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.531 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.524+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.532 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.524+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.532 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.525+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.532 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.525+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.535 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.525+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.535 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.525+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.535 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.525+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.535 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.526+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.535 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.526+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.536 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.527+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.536 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.527+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.536 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.528+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.536 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.529+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.536 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.529+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.536 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.529+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.536 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.529+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.536 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.531+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.537 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.532+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.537 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.532+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.537 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.533+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.538 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.534+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.538 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.534+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.538 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.534+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.539 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.536+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.539 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.536+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.539 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.538+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.540 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.538+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.540 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.539+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.541 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.541+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.542 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.542+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.543 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.542+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.548 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.543+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.548 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.544+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.548 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.545+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.554 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.545+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.554 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.545+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.554 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.546+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.554 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.546+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.554 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.546+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.554 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.546+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.554 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.547+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.547+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.548+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.548+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.549+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.550+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.551+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.551+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.552+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.552+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.553+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.553+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.553+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.553+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.555 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.553+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.556 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.554+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.556 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.554+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.556 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.555+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.556 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.556+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.557 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.556+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.557 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.556+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.558 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.557+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.558 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.558+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.558 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.558+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.559 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.559+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.560 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.560+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.560 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.560+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.561 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.560+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.561 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.560+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.565 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.561+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.565 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.561+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.565 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.562+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.565 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.563+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.565 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.563+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.565 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.564+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.565 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.564+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.565 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.564+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.567 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.565+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.567 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.565+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.567 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.566+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.567 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.566+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.571 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.569+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.571 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.569+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.573 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.570+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.573 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.570+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.573 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.570+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.574 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.570+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.574 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.570+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.574 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.570+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.574 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.573+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.574 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.573+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.574 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.573+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.574 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.574+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.575 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.575+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.575 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.575+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.576 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.575+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.576 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.576+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.576 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.576+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.578 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.577+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.578 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.577+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.578 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.578+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.579 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.578+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.580 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.579+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.583 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.580+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.584 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.581+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.584 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.582+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.584 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.582+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.584 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.582+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.584 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.582+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.584 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.582+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.585 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.584+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.585 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.584+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.587 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.586+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.588 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.587+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.589 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.588+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.590 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.588+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.591 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.591+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.594 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.591+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.594 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.591+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.595 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.593+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.595 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.594+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.595 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.594+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.595 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.594+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.595 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.594+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.597 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.595+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.597 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.596+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.597 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.596+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.597 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.596+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.597 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.597+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.601 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.598+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.601 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.598+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.605 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.602+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.608 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.602+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.608 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.603+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.608 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.603+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.608 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.604+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.610 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.605+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.610 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.606+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.610 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.606+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.610 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.606+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.610 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.607+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.610 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.607+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.610 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.607+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.611 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.608+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.611 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.609+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.611 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.609+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.611 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.610+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.614 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.611+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.614 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.612+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.615 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.612+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.615 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.613+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.615 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.613+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.615 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.614+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.615 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.614+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.615 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.614+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.615 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.615+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.616 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.615+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.616 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.615+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.616 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.616+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.617 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.616+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.617 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.617+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.617 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.617+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.619+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.619+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.620+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.620+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.620+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.621+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.623 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.621+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.625 INFO:teuthology.orchestra.run.smithi107.stdout: 16 16 15310 15294 3.73328 3.01953 0.0153616 0.0167272 2021-02-12T17:59:08.625 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.622+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.626 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.623+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.626 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.623+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.626 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.623+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.626 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.625+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.628 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.627+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.628 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:08.628+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:08.631 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.627+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.631 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.627+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.631 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.627+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.631 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.628+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.631 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.630+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.632 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.630+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.632 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.630+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.632 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.631+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.633 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.632+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.633 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.633+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.637 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.634+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.637 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.634+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.637 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.634+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.637 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.634+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.638 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.634+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.638 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.635+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.638 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.635+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.638 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.636+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.639 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.637+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.639 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.638+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.639 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.639+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.641 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.639+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.642 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.640+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.642 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.642+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.643 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.642+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.643 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.642+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.643 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.642+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.643 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.642+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.644 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.643+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.644 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.643+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.644 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.644+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.645 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.645+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.645 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.645+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.646 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.645+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.646 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.645+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.647 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.646+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.647 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.646+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.647 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.647+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.648 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.648+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.649 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.648+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.649 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.648+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.650 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.649+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.650 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.650+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.652 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.651+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.652 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.652+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.653 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.652+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.653 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.652+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.655 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.653+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.655 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.653+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.656 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.654+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.656 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.655+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.657 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.655+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.657 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.656+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.658 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.658+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.661 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.658+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.661 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.658+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.661 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.659+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.661 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.659+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.662 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.660+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.662 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.660+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.662 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.660+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.662 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.660+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.662 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.661+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.662 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.661+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.663 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.662+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.663 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.662+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.664 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.663+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.664 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.663+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.665 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.665+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.666 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.666+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.667 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.667+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.668 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.667+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.668 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.668+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.668 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.668+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.669 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.668+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.670 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.669+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.670 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.670+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.670 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.670+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.671 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.670+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.672 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.671+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.672 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.671+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.672 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.672+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.672 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.672+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.673 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.672+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.673 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.673+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.673 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.673+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.675 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.674+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.676 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.675+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.676 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.676+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.677 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.677+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.678 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.677+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.678 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.677+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.679 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.679+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.682 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.679+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.686 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.680+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.686 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.682+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.688 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.682+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.689 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.683+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.689 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.684+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.689 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.685+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.690 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.685+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.690 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.685+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.690 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.687+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.691 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.688+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.691 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.688+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.691 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.688+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.692 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.689+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.692 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.690+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.692 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.691+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.693 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.691+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.693 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.691+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.693 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.691+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.695 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.694+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.695 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.694+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.695 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.694+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.695 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.695+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.695 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.695+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.698 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.697+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.698 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.698+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.698 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.698+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.699 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.698+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.699 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.699+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.700 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.700+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.701 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.700+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.703 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.702+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.704 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.704+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.704 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.704+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.705 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.705+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.705 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.705+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.707 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.706+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.707 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.706+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.707 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.706+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.707 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.706+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.707 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.707+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.707 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.707+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.708 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.708+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.708 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.708+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.708 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.708+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.709 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.708+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.709 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.709+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.714 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.712+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.716 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.713+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.716 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.713+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.716 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.713+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.716 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.715+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.716 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.715+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.716 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.716+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.717 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.716+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.717 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.716+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.719 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.718+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.719 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.719+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.726 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.720+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.727 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.722+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.727 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.722+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.728 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.725+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.728 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.725+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.728 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.725+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.728 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.726+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.728 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.726+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.729 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.727+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.729 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.727+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.729 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.728+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.729 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.728+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.729 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.728+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.729 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.729+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.729 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.729+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.730 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.729+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.732 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.730+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.732 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.731+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.733 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.732+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.733 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.732+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.733 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.732+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.733 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.732+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.733 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.733+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.736 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.734+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.736 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.734+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.736 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.735+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.736 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.736+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.737 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.736+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.737 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.736+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.737 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.737+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.738 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.738+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.739 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.739+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.741 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.741+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.741 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.741+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.742 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.742+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.742 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.742+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.742 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.742+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.743 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.743+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.744 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.743+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.744 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.743+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.744 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.743+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.744 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.743+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.745 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.744+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.745 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.744+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.746 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.746+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.747 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.747+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.747 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.747+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.748 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.747+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.750 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.748+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.750 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.748+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.750 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.749+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.751 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.749+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.752 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.750+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.752 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.750+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.755 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.751+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.755 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.752+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.755 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.752+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.755 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.754+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.755 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.754+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.756 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.755+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.756 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.755+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.756 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.755+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.756 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.756+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.757 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:08.757+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:08.758 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.758+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.759 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.759+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.759 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.759+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.759 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.759+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.761 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.761+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.762 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.761+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.763 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.763+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.766 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.763+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.766 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.763+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.766 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.763+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.766 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.765+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.767 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.766+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.767 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.766+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.767 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.767+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.768 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.768+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.769 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.768+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.769 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.769+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.769 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.769+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.769 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.769+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.769+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.769+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.770 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.770+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.771 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.770+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.771 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.771+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.773 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.772+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.773 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.772+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.773 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.773+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.777 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.775+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.777 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.775+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.781 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.776+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.783 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.777+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.783 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.777+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.783 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.777+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.783 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.777+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.784 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.778+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.786 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.778+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.786 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.778+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.786 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.779+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.786 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.779+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.786 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.780+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.780+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.781+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.781+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.782+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.782+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.783+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.784+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.787 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.784+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.788 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.786+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.788 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.786+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.788 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.786+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.788 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.787+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.789 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.788+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.789 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.788+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.789 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.788+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.789 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.789+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.789 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.789+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.792 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.790+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.792 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.790+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.792 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.791+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.792 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.791+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.792 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.791+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.792 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.791+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.801 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.792+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.801 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.793+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.802 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.795+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.806 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.796+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.806 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.796+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.797+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.797+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.799+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.799+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.800+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.800+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.800+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.807 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.800+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.802+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.802+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.802+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.805+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.805+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.805+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.808 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.806+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.807+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.808+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.808+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.808+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.809 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.809+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.813 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.809+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.813 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.811+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.814 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.811+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.814 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.814+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.814 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.814+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.815 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.814+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.815 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.815+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.816 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.815+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.817+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.819 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.817+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.820 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.819+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.820 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.819+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.820 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.819+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.820 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.819+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.821 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.820+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.821 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.821+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.822 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.821+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.822 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.822+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.823 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.823+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.823 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.823+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.825 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.824+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.825 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.824+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.826 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.825+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.826 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.825+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.826 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.826+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.826 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.826+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.826 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.826+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.827 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.826+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.831 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.827+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.831 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.828+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.831 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.828+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.831 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.829+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.831 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.829+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.831 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.830+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.832 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.831+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.832 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.831+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.832 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.832+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.834 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.834+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.834 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.834+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.834+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.834+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.835 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.835+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.836 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.836+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.837 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.837+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.840 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.837+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.840 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.837+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.841 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.838+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.841 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.838+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.841 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.838+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.841 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.839+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.839+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.839+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.839+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.840+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.842 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.841+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.843 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.842+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.843 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.843+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.844 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.843+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.844 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.843+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.844 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.844+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.845 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.844+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.849 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.846+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.849 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.846+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.849 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.847+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.849 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.848+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.850 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.849+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.850 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.849+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.850 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.849+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.851 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.850+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.851 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.851+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.853 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.852+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.853 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.852+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.853 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.853+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.854 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.854+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.855 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.854+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.856 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.855+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.856 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.856+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.858 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.858+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.858 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.858+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.859 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.858+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.861 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.859+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.862 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.859+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.862 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.859+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.862 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.860+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.862 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.860+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.862 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.860+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.862 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.862+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.863 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.862+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.863 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.862+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.863 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.863+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.863 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.863+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.864 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.864+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.865 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.864+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.865 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:08.865+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:08.866 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.865+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.866 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.866+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.867 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.867+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.867 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.867+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.873 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.869+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.873 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.870+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.873 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.870+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.873 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.870+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.873 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.871+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.871+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.872+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.872+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.873+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.874 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.873+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.876 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.873+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.878 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.875+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.878 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.875+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.878 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.876+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.877+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.877+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.877+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.877+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.878+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.878+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.878+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.880 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.879+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.880 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.879+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.880 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.880+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.881 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.881+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.882 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.881+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.890 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.884+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.890 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.885+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.890 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.886+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.891 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.887+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.891 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.887+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.892 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.891+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.892 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.891+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.893 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.891+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.893 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.891+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.893 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.892+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.893 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.892+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.893 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.893+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.893 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.893+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.894 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.893+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.894 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.894+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.895 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.894+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.895 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.895+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.897 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.897+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.898+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.898+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.898+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.899+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.899+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.900+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.901 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.900+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.903 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.901+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.907 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.903+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.907 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.904+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.907 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.904+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.907 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.904+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.908 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.905+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.908 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.905+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.908 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.905+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.908 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.906+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.908 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.906+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.909 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.907+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.909 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.907+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.909 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.907+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.909 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.908+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.909 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.909+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.910 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.909+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.910 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.910+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.911 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.910+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.911 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.910+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.912 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.912+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.913 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.912+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.913 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.913+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.915 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.914+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.915 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.915+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.915 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.915+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.916 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.915+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.916 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.916+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.917 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.916+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.919 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:08.918+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:08.919 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.918+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.919 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.918+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.920 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.919+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.920 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.919+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.923 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.922+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.923 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.922+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.923 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.922+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.924 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.923+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.924 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.923+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.924 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.924+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.924 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.924+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.925 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.925+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.926+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.926+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.927+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.927+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.927+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.929 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.927+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.930 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.930+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.931 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.930+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.931 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.931+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.934 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.931+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.937 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.933+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.937 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.933+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.937 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.933+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.938 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.934+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.938 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.936+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.938 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.936+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.938 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.936+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.939 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.936+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.939 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.936+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.939 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.937+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.939 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.937+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.939 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.937+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.941 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.941+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.941 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.941+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.942+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.942 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.942+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.944 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.942+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.944 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.943+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.945 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.943+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.945 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.944+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.947 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.944+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.947 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.944+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.947 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.945+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.948 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.946+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.948 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.947+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.949 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.949+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.950 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.950+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.952 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.951+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.955 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.947+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.955 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.952+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.956 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.952+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.952+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.953+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.953+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.954+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.954+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.954+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.955+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.955+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.957 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.956+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.958 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.957+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.958 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.957+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.958 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.958+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.960 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.958+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.961 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.959+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.961 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.960+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.962 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.960+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.962 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.960+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.962 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.962+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.963 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.962+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.965 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.964+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.965 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.964+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.965 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.964+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.965 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.965+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.966 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.966+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.966 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.966+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.967 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.967+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.972 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.967+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.972 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.967+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.973 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.969+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.973 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.970+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.973 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.970+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.974 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.971+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.974 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.971+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.974 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.972+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.974 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.972+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.974 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.972+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.975 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.973+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.975 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.974+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.975 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.974+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.975 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.975+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.976 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.975+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.975+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.976+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.976+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.977+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.977+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.977+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.978 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.978+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.981 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.981+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.982 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.981+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.986 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.983+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.986 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.983+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.986 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.983+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.986 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.983+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.986 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.984+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.986 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.985+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.986 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.985+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.987 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.986+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.987 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.986+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.987 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.987+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.987+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.988+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.988+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.989+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.989+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.989+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.990+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.992 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.990+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.990+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.991+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.992+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.993 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.993+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.994+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.994+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.995 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.995+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.996 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.995+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.996 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.996+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.997 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.996+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.997 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.997+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.998 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.997+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:08.998 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.997+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.000 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.998+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.001 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.998+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.001 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.999+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.001 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:08.999+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.001 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.000+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.001 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.000+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.002 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.000+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.002 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.000+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.002 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.001+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.002 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.001+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.003 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.003+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.004+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.007 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.004+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.008 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.005+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.008 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.006+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.008 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.006+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.009 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.006+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.009 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.007+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.009 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.008+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.010 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.008+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.010 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.008+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.010 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.008+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.010 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.008+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.010 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.008+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.010 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.009+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.011 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.010+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.012 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.012+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.012 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.012+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.013 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.012+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.014 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.014+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.016 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.015+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.018 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.018+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.021 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.018+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.021 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.019+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.021 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.019+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.021 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.020+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.021 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.020+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.021 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.021+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.023 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.022+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.023 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.022+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.023 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.023+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.026+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.026+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.026+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.026+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.027+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.027+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.028+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.028+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.032 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.029+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.033 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.029+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.033 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.029+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.034 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.030+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.030+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.031+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.031+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.031+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.031+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.031+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.032+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.035 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.033+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.034+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.036 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.035+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.038 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.036+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.039 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.037+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.039 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.037+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.039 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.038+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.039 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.038+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.040 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.039+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.047 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.041+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.047 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.041+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.048 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.043+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.048 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.043+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.048 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.044+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.049 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.046+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.049 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.047+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.049 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.047+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.049 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.047+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.050 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.048+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.050 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.049+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.050 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.049+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.050 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.049+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.051 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.050+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.051 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.050+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.051 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.050+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.051 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.051+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.052 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.051+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.057 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.053+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.057 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.054+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.058 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.054+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.058 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.054+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.058 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.055+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.058 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.055+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.058 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.056+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.058 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.056+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.057+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.057+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.057+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.059 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.058+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.060 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.058+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.060 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.059+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.061 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.060+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.064 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.060+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.064 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.060+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.061+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.061+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.062+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.068 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.063+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.063+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.063+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.064+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.064+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.064+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.065+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.069 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.065+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.070 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.065+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.071 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.066+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.071 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.067+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.071 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.067+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.071 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.068+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.071 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.068+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.072 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.069+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.073 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.071+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.073 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.071+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.073 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.072+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.073 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.072+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.074 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.073+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.074 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.074+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.075 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.074+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.075 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.075+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.076 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.075+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.076 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.076+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.076 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.076+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.077 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.076+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.077 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.077+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.077 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.077+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.078 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.078+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.079 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.078+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.079 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.079+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.079 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.079+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.080 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.079+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.080 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.080+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.080 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.080+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.081+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.081+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.081+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.083 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.082+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.084 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.082+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.084 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.082+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.084 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.082+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.085 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.083+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.085 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.083+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.085 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.083+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.085 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.083+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.085 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.083+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.085 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.084+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.085 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.084+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.086 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.084+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.086 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.084+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.087 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.086+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.087 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.086+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.088 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.088+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.090 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.089+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.090 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.089+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.092 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.091+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.092 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.091+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.092 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.092+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.093 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.092+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.093+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.095 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.094+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.096 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.094+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.097 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.095+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.098 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.096+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.098 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.096+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.099 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.097+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.099 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.098+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.099 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.098+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.100 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.098+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.100 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.099+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.100 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.099+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.100 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.099+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.101 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.099+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.102 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.099+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.102 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.100+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.102 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.101+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.103 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.102+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.108 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.103+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.108 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.103+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.109 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.105+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.109 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.105+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.109 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.105+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.109 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.106+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.109 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.107+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.109 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.109+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.110 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.109+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.110 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.110+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.110 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.110+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.112 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.111+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.114 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.112+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.116 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.114+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.116 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.114+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.116 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.115+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.116 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.115+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.117 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.116+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.117 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.116+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.117 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.117+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.117 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.117+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.118 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.117+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.118 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.117+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.118 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.118+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.119 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.118+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.119 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.118+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.119 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.119+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.119 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.119+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.120 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.119+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.120 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.120+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.120 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.120+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.120+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.121+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.121 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.121+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.122 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.121+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.126 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.123+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.126 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.123+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.126 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.124+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.126 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.125+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.126 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.125+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.126 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.125+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.127 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.126+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.127 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.127+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.127 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.127+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.132 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.129+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.132 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.130+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.132 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.130+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.134 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.130+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.134 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.130+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.134 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.131+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.134 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.131+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.134 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.132+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.134 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.132+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.135 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.133+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.135 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.133+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.135 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.133+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.135 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.134+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.135 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:09.135+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:09.135 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.134+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.136 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.135+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.137 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.137+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.139 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.138+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.139 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.138+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.140 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.139+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.141 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.140+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.141 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.141+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.142 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.142+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.143 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.142+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.144 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.143+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.144 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.143+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.144 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.144+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.144 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.144+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.146 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.145+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.146 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.145+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.146 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.145+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.148 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.146+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.148 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.148+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.149 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.149+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.149 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.149+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.149 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.149+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.154 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.151+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.154 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.153+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.154 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.153+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.154 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.154+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.156 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.155+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.156 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.155+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.156 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.155+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.157 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.157+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.158+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.158+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.159+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.159 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.159+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.160 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.159+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.160 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.160+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.161 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.161+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.161 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.161+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.162 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.162+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.162 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.162+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.163 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.163+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.163 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.163+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.167 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.164+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.167 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.165+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.167 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.165+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.167 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.166+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.167 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.167+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.168 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.167+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.168 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.168+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.171 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.170+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.171 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.171+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.172 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.171+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.172 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.171+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.172 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.172+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.173 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.173+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.174 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.173+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.175 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.174+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.175 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.175+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.176 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.175+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.176 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.175+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.176 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.176+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.177 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.176+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.177 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.177+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.177 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.177+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.179 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.177+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.179 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.178+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.179 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.178+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.180 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.179+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.180 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.179+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.180 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.179+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.180 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.179+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.180 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.180+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.181 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.181+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.182 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.182+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.183 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.183+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.184 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.184+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.184 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.184+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.188 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.187+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.191 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.188+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.191 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.188+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.192 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.190+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.192 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.190+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.192 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.191+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.192 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.191+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.192 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.191+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.194 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.194+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.195 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.195+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.195 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.195+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.196 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.195+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.196 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.195+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.200 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.197+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.200 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.197+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.201 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.198+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.201 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.198+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.201 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.198+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.201 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.200+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.202 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.201+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.202 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.202+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.203 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:09.202+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:09.203 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.203+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.203 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.203+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.204 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.203+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.204 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.204+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.206 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.206+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.208 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.207+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.208 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.207+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.208 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.208+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.208 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.208+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.208+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.208+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.209+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.210+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.210+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.210+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.210+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.211+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.211+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.211+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.212+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.213+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.213+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.215+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.215+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.216+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.216+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.217+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.218+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.218+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.219+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.219+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.219+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.220+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.221+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.222+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.223+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.224+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.225+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.225+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.225+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.225+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.226+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.226+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.227+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.227+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.228+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.229+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.230+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.231+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.232+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.234+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.235+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.235+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.236+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.236+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.236+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.238+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.238+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.238+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.238+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.239+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.239+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.240+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.240+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.240+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.243+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.243+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.245+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.245+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.246+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.247+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.247+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.247+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.247+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.249+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.249+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.250+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.250+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:09.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.252+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.123 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.253+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.123 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.254+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.123 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.255+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.123 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2021-02-12T17:59:10.123 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.255+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.256+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.256+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.257+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.258+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.259+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.260+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.262+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.262+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.209 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.262+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.262+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.262+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.263+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.263+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.264+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.264+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.265+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.266+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.268+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.268+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.268+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.269+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.210 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.269+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.270+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.270+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.270+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.271+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.272+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.273+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.273+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.273+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.274+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.275+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.275+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.276+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.276+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.277+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.211 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.278+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.279+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.279+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.280+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.281+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.281+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.281+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.281+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.282+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.282+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.282+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.283+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.283+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.284+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.286+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.212 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.286+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.286+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.288+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.288+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.289+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.289+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.289+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.290+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.291+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.291+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.292+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.292+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.294+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.295+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.213 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.295+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.295+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.296+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.296+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.297+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.297+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.297+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.297+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.298+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.298+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.298+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.299+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.300+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.214 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.300+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.301+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.301+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.303+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.304+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.304+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.305+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.305+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.305+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.306+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.306+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.307+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.308+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.308+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.308+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.215 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.309+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.309+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.309+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.310+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.311+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.312+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.312+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.317+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.317+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.317+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.319+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.319+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.319+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.320+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.216 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.320+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.320+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.321+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.321+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.322+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.323+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.323+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.325+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.325+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.325+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.325+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.325+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.326+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.328+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.329+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.329+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.217 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.330+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.330+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.331+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.332+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.332+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.332+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.333+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.333+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.335+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.335+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.337+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.337+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.339+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.340+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.218 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.340+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.340+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.340+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.341+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.341+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.342+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.343+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.343+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.343+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.343+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.343+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.344+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.345+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.346+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.219 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.347+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.347+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.347+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.348+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.348+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.348+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.349+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.353+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.353+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.353+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.355+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.356+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.356+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.358+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.359+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.220 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.359+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.360+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.360+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.364+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.364+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.365+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.365+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.366+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.366+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.368+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.368+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.368+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.369+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.369+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.221 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.369+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.370+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.370+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.370+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.371+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.371+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.371+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.372+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.373+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.373+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.374+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.374+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.375+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.222 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.375+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.376+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.377+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.377+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.379+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.380+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.380+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.380+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.380+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.381+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.382+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.383+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.383+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.384+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.223 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.384+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.384+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.385+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.385+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.385+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.386+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.387+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.387+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.387+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.389+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.389+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.391+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.393+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.393+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.224 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.394+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.394+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.395+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.395+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.397+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.397+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.398+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.398+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.399+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.400+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.400+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.400+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.400+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.400+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.400+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.225 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.402+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.402+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.405+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.405+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.406+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.407+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.408+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.409+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.409+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.410+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.410+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.412+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.413+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.226 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.413+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.413+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.414+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.416+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.416+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.416+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.416+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.418+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.419+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.419+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.420+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.420+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.420+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.421+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.227 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.421+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.422+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.422+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.423+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.423+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.424+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.425+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.425+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.427+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.427+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.428+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.429+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.430+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.228 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.431+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.431+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.431+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.432+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.432+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.432+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.432+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.433+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.434+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.434+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.435+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.435+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.435+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.436+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.229 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.438+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.438+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.438+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.438+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.439+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.440+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.440+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.441+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.441+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.441+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.442+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.442+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.230 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.443+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.443+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.443+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.443+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.444+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.445+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.445+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.446+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.446+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.446+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.447+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.447+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.448+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.448+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.231 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.449+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.449+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.450+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.451+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.451+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.453+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.453+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.453+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.454+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.455+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.456+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.457+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.232 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.457+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.457+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.459+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.459+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.460+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.460+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.460+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.461+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.461+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.462+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.462+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.464+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.464+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.233 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.465+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.466+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.466+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.466+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.467+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.467+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.468+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.471+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.472+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.472+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.473+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.473+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.234 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.473+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.473+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.474+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.474+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.474+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.475+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.476+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.477+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.477+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.478+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.478+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.478+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.481+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.235 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.481+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.483+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.484+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.484+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.485+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.485+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.486+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.486+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.486+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.486+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.487+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.488+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.488+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.236 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.489+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.489+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.489+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.490+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.490+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.491+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.492+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.492+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.493+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.493+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.493+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.495+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.496+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.237 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.497+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.497+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.498+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.498+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.499+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.499+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.500+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.500+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.500+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.501+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.501+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.502+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.502+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.504+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.238 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.507+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.507+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.507+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.509+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.510+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.510+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.510+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.510+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.513+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.513+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.514+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.514+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.514+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.239 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.514+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.515+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.515+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.515+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.516+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.517+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.518+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.518+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.518+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.519+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.520+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.521+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.522+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.524+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.524+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.240 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.525+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.526+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.526+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.527+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.527+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.528+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.528+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.528+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.528+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.529+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.530+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.531+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.532+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.241 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.532+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.533+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.533+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.535+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.535+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.535+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.535+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.537+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.537+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.537+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.537+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.538+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.538+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.242 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.539+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.539+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.540+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.540+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.541+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.541+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.541+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.542+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.545+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.545+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.545+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.546+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.546+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.548+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.549+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.243 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.549+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.550+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.550+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.551+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.552+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.554+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.554+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.554+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.555+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.556+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.556+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.557+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.557+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.557+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.244 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.558+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.558+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.560+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.560+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.560+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.560+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.561+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.561+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.561+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.563+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.563+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.563+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.564+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.245 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.564+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.565+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.565+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.566+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.567+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.567+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.568+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.568+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.569+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.569+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.570+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.570+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.246 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.570+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.571+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.572+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.573+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.574+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.574+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.574+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.575+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.575+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.579+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.580+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.580+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.581+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.582+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.247 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.582+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.583+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.585+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.585+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.586+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.586+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.586+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.586+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.587+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.587+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.587+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.587+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.587+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.248 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.587+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.589+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.589+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.590+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.592+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.593+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.595+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.595+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.596+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.596+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.597+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.597+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.597+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.598+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.598+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.249 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.598+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.600+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.601+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.602+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.602+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.602+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.603+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.604+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.604+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.605+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.605+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.606+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.606+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.250 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.606+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.607+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.607+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.607+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.607+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.608+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.609+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.609+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.609+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.610+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.612+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.613+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.613+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.614+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.615+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.251 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.615+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.615+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.615+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.616+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.616+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.616+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.618+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.618+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.619+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.620+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.621+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.622+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.623+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.252 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.623+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.624+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.624+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.625+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.627+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.628+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.628+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.628+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.630+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.630+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.631+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.631+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.632+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.632+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.633+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.253 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.633+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.634+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.634+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.635+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.636+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.638+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.639+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.640+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.640+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.642+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.642+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.642+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.642+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.254 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.644+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.645+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.645+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.645+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.645+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.646+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.646+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.646+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.647+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.647+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.649+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.649+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.649+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.650+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.650+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.255 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.651+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.652+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.652+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.653+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.653+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.653+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.655+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.656+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.656+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.657+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.658+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.658+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.658+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.256 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.659+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.659+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.659+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.659+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.660+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.660+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.660+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.661+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.661+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.661+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.662+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.663+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.663+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.665+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.257 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.668+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.668+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.668+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.668+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.669+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.670+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.671+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.671+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.671+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.672+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.673+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.673+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.673+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.258 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.674+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.675+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.675+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.677+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.677+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.677+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.679+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.680+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.680+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.680+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.680+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.682+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.682+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.682+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.259 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.683+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.683+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.684+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.684+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.685+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.685+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.685+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.685+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.686+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.687+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.687+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.688+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.688+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.690+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.260 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.690+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.691+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.692+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.693+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.694+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.694+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.694+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.694+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.695+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.695+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.696+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.697+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.697+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.698+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.699+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.261 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.700+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.700+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.701+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.701+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.701+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.702+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.702+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.703+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.703+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.706+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.707+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.707+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.707+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.262 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.708+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.708+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.709+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.710+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.710+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.710+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.711+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.711+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.711+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.713+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.713+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.714+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.714+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.263 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.716+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.716+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.716+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.717+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.718+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.718+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.718+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.719+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.720+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.721+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.722+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.722+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.722+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.723+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.264 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.726+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.726+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.726+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.727+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.728+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.728+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.729+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.729+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.729+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.729+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.729+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.732+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.733+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.265 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.733+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.734+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.735+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.735+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.735+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.736+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.737+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.737+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.738+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.738+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.738+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.739+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.740+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.266 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.740+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.741+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.743+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.744+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.744+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.745+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.746+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.746+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.748+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.748+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.749+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.749+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.750+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.751+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.267 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.751+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.752+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.752+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.752+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.752+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.752+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.754+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.754+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.754+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.755+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.755+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.758+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.758+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.759+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.268 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.759+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.760+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.760+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.761+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.762+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.762+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.764+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.764+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.766+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.766+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.767+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.768+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.768+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.269 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.768+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.769+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.770+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.770+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.770+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.771+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.771+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.772+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.774+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.774+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.774+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.775+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.270 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.775+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.775+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.776+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.777+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.777+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.778+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.779+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.779+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.779+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.780+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.781+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.781+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.783+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.783+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.271 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.785+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.785+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.786+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.787+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.787+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.788+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.788+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.789+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.790+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.792+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.793+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.794+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.794+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.272 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.795+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.795+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.796+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.798+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.798+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.798+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.799+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.799+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.799+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.799+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.800+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.801+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.801+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.273 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.801+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.801+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.802+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.803+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.803+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.804+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.804+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.804+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.804+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.805+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.805+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.805+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.806+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.274 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.806+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.806+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.807+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.807+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.807+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.807+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.808+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.809+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.810+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.811+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.811+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.811+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.811+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.814+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.275 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.814+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.815+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.815+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.816+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.817+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.817+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.818+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.819+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.820+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.820+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.821+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.821+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.822+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.822+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.276 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.823+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.823+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.824+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.824+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.824+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.825+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.827+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.828+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.828+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.828+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.829+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.829+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.830+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.277 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.833+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.833+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.833+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.833+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.834+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.835+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.835+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.836+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.836+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.838+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.838+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.839+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.839+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.278 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.839+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.839+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.839+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.840+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.841+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.841+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.841+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.842+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.843+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.843+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.844+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.844+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.845+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.846+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.279 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.846+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.847+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.847+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.847+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.848+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.850+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.850+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.851+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.851+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.852+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.852+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.852+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.853+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.280 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.853+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.854+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.854+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.855+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.855+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.856+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.856+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.859+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.860+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.860+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.861+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.862+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.862+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.863+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.281 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.863+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.866+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.866+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.866+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.866+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.866+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.868+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.868+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.868+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.869+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.870+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.870+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.870+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.870+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.282 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.871+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.871+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.871+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.872+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.873+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.873+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.873+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.874+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.874+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.874+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.875+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.876+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.876+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.877+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.878+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.283 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.878+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.881+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.881+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.881+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.882+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.882+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.883+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.883+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.886+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.886+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.886+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.886+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.889+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.889+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.284 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.889+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.890+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.890+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.891+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.892+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.892+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.893+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.893+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.894+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.896+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.896+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.897+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.897+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.898+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.898+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.285 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.899+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.899+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.899+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.899+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.901+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.903+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.903+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.904+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.904+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.905+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.906+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.906+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.907+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.286 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.907+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.908+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.908+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.908+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.908+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.909+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.910+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.911+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.911+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.911+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.912+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.912+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.913+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.914+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.916+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.287 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.916+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.916+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.916+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.917+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.917+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.918+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.919+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.919+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.920+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.920+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.921+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.922+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.922+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.923+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.288 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.923+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.923+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.927+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.927+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.927+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.929+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.931+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.931+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.932+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.932+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.933+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.933+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.933+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.289 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.934+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.936+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.936+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.936+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.938+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.938+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.938+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.939+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.939+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.939+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.939+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.939+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.941+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.290 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.941+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.942+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.942+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.945+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.946+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.946+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.946+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.947+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.947+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.948+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.949+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.949+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.949+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.949+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.952+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.291 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.952+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.953+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.954+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.954+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.955+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.956+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.956+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.956+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.956+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.957+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.958+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.958+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.959+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.960+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.963+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.292 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.964+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.964+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.964+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.964+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.965+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.965+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.965+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.967+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.967+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.968+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.968+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.969+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.970+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.970+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.971+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.293 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.973+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.974+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.975+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.975+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.976+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.976+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.976+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.977+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.977+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.978+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.979+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.979+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.980+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.294 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.980+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.980+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.980+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.980+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.982+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.982+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.983+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.983+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.984+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.984+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.985+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.985+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.986+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.986+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.987+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.295 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.987+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.988+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.988+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.989+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.989+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.990+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.990+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.990+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.991+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.991+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.991+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.991+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.993+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.993+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.994+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.296 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.995+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.996+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.996+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.996+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.998+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.999+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.999+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.999+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:09.999+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.000+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.000+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.001+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.001+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.003+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.297 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.003+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.004+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.005+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.005+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.005+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.006+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.006+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.006+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.006+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.006+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.008+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.009+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.009+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.009+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.298 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.012+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.012+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.012+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.015+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.016+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.016+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.016+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.017+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.017+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.017+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.018+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.018+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.019+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.019+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.020+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.299 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.021+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.021+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.021+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.022+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.023+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.023+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.024+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.024+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.025+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.025+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.025+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.025+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.026+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.300 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.026+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.027+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.027+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.028+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.028+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.029+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.029+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.029+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.030+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.030+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.031+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.032+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.032+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.032+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.301 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.033+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.033+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.034+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.035+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.035+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.035+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.036+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.037+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.037+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.038+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.040+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.040+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.041+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.041+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.302 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.042+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.044+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.044+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.047+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.047+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.047+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.047+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.048+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.049+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.049+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.050+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.051+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.052+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.052+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.303 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.053+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.053+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.054+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.055+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.056+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.057+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.057+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.058+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.061+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.061+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.062+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.062+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.063+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.064+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.065+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.304 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.065+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.066+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.066+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.066+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.067+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.068+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.069+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.069+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.070+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.070+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.071+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.072+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.072+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.305 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.073+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.073+0000 7fb8dd2ee700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.074+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.074+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.075+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.075+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.075+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.076+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.076+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.078+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.078+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.079+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.079+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.306 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.079+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.080+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.082+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.082+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.083+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.083+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.084+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.084+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.085+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.085+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.085+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.086+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.087+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.087+0000 7fb8dcaed700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.307 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.087+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.088+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.088+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.091+0000 7fb8e32fa700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.092+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.092+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.092+0000 7fb8de2f0700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.092+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.093+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.093+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.093+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.093+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.094+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.094+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.308 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.095+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.096+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.096+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.097+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.098+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.100+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.101+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.101+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.102+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.102+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.103+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.103+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.103+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.103+0000 7fb8e0af5700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.104+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.309 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.105+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.105+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.106+0000 7fb8deaf1700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.106+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.107+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.107+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.108+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.108+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.109+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.110+0000 7fb8e12f6700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.110+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.110+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.112+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.112+0000 7fb8df2f2700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.310 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.113+0000 7fb8dc2ec700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.113+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.113+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.114+0000 7fb8e3afb700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.115+0000 7fb8e2af9700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.116+0000 7fb8e02f4700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.116+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.116+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.117+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.117+0000 7fb8dfaf3700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.118+0000 7fb8e1af7700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.118+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.119+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.121+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.121+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.311 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.121+0000 7fb8ddaef700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.312 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.121+0000 7fb8e22f8700 -1 bdev(0x55f8aaebe400 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.312 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.122+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.312 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.122+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.312 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.123+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.312 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:10.123+0000 7fb8f10bb700 -1 bdev(0x55f8aaebe800 /var/lib/ceph/osd/ceph-1/block) aio_write objectstore_blackhole=true, throwing out IO 2021-02-12T17:59:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:09.332+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:09.357+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:09.374+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:09.578+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:09.608+0000 7febdc622700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:09.723+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:10.056+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x555983091860 legacy=0x7febc803ab40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:10.312 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:10.080+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6800/33987 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:10.313 INFO:teuthology.orchestra.run.smithi107.stdout: 17 16 16171 16155 3.71086 3.36328 0.023275 0.0168275 2021-02-12T17:59:10.313 INFO:tasks.ceph.osd.1:Stopped 2021-02-12T17:59:10.627 INFO:teuthology.orchestra.run.smithi107.stdout: 18 16 16604 16588 3.59868 1.69141 0.023119 0.0168736 2021-02-12T17:59:11.581 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:11.580+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:11.627 INFO:teuthology.orchestra.run.smithi107.stdout: 19 16 16765 16749 3.44241 0.628906 0.00564792 0.0181478 2021-02-12T17:59:11.870 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:11.869+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.089 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.088+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.205 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.205+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.313 INFO:tasks.ceph.osd.1:Restarting daemon 2021-02-12T17:59:12.314 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2021-02-12T17:59:12.316 INFO:tasks.ceph.osd.1:Started 2021-02-12T17:59:12.316 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2021-02-12T17:59:12.351 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.350+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.369 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.369+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.462 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.460+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2021-02-12T17:59:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:0 2021-02-12T17:59:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:1 2021-02-12T17:59:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:2 2021-02-12T17:59:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2021-02-12T17:59:12.502 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2021-02-12T17:59:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:config diff 2021-02-12T17:59:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:config diff get 2021-02-12T17:59:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2021-02-12T17:59:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:config help [] 2021-02-12T17:59:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2021-02-12T17:59:12.503 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2021-02-12T17:59:12.505 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-12T17:59:12.505 INFO:tasks.osd_backfill.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2021-02-12T17:59:12.527 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.526+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.621 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.620+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:12.627 INFO:teuthology.orchestra.run.smithi107.stdout:2021-02-12T17:59:12.627954+0000 min lat: 0.00418347 max lat: 1.41286 avg lat: 0.0174681 2021-02-12T17:59:12.628 INFO:teuthology.orchestra.run.smithi107.stdout: sec Cur ops started finished avg MB/s cur MB/s last lat(s) avg lat(s) 2021-02-12T17:59:12.628 INFO:teuthology.orchestra.run.smithi107.stdout: 20 16 18332 18316 3.57629 6.12109 0.00763149 0.0174681 2021-02-12T17:59:12.651 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.651+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.725 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.724+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.806 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.805+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.856 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.855+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.910 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.910+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:12.911 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.910+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:12.953 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:12.953+0000 7f64bc5b9080 -1 Falling back to public interface 2021-02-12T17:59:12.982 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.981+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:12.993 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:12.992+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.277+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.278 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.277+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:13.331 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.330+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.373 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.372+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.376 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.375+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.614 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.613+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.627 INFO:teuthology.orchestra.run.smithi107.stdout: 21 16 19910 19894 3.69947 6.16406 0.0102595 0.0168882 2021-02-12T17:59:13.660 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.659+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.692 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.692+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:13.776 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.775+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.800 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.799+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.828 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.827+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.831 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.831+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.838 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.837+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:13.856 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.856+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:13.990 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:13.989+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.060 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.060+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.415 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.415+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.460 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.460+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.534 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.533+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.625 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.624+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.625 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.625+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.628 INFO:teuthology.orchestra.run.smithi107.stdout: 22 16 21424 21408 3.79999 5.91406 0.0176167 0.0164419 2021-02-12T17:59:14.702 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.701+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:14.717 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.717+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.757 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.756+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:14.822 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:14.821+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:15.299 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:15.299+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:15.300 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:15.299+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:15.628 INFO:teuthology.orchestra.run.smithi107.stdout: 23 16 22810 22794 3.87014 5.41406 0.0200757 0.0161417 2021-02-12T17:59:15.873 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:15.872+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.384 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.384+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.390 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.389+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.429 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.429+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.430 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.429+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:16.455 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.455+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.545 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.544+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.628 INFO:teuthology.orchestra.run.smithi107.stdout: 24 16 24158 24142 3.92825 5.26562 0.00834052 0.015901 2021-02-12T17:59:16.629 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.628+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.717 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.716+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.756 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.756+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.793 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.792+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.843 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.842+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:16.910 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:16.909+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:17.018 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:17.017+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:17.059 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:17.059+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:17.279 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:17.279+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:17.339 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:17.338+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:17.440 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:17.439+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:17.506 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2021-02-12T17:59:17.628 INFO:teuthology.orchestra.run.smithi107.stdout: 25 16 25259 25243 3.94314 4.30078 0.0181192 0.015843 2021-02-12T17:59:17.672 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:17.671+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:17.714 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2021-02-12T17:59:17.715 INFO:teuthology.orchestra.run.smithi107.stderr:0 2021-02-12T17:59:17.715 INFO:teuthology.orchestra.run.smithi107.stderr:1 2021-02-12T17:59:17.715 INFO:teuthology.orchestra.run.smithi107.stderr:2 2021-02-12T17:59:17.715 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2021-02-12T17:59:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2021-02-12T17:59:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2021-02-12T17:59:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2021-02-12T17:59:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2021-02-12T17:59:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator fragmentation block 2021-02-12T17:59:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator score block 2021-02-12T17:59:17.716 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2021-02-12T17:59:17.717 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-12T17:59:17.718 INFO:tasks.osd_backfill.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2021-02-12T17:59:17.913 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:17.912+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:18.210 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:18.210+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:18.427 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:18.426+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:18.534 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:18.534+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:18.628 INFO:teuthology.orchestra.run.smithi107.stdout: 26 16 26547 26531 3.98497 5.03125 0.0135193 0.0156783 2021-02-12T17:59:18.646 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:18.645+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:18.762 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:18.761+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:18.934 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:18.934+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:19.123 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:19.123+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:19.146 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:19.146+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:19.234 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:19.233+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:19.629 INFO:teuthology.orchestra.run.smithi107.stdout: 27 16 27799 27783 4.01849 4.89062 0.0148657 0.0155471 2021-02-12T17:59:19.819 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:19.818+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:19.846 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:19.846+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:20.014 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:20.013+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:20.366 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:20.365+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:20.513 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:20.512+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:20.539 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:20.538+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:20.594 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:20.593+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:20.616 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:20.616+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:20.628 INFO:teuthology.orchestra.run.smithi107.stdout: 28 16 28981 28965 4.03986 4.61719 0.00779538 0.0154621 2021-02-12T17:59:21.069 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.069+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:21.093 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.093+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:21.136 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.136+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:21.242 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.242+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:21.269 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.269+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1)._try_send injecting socket failure 2021-02-12T17:59:21.364 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.364+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:21.629 INFO:teuthology.orchestra.run.smithi107.stdout: 29 16 30115 30099 4.05328 4.42969 0.0163658 0.0154125 2021-02-12T17:59:21.667 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.666+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:21.667 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.667+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:21.818 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.817+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:21.982 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:21.982+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:22.028 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:22.028+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x7febc8035610 legacy=0x7febc8042130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:22.588 INFO:teuthology.orchestra.run.smithi107.stderr:2021-02-12T17:59:22.587+0000 7febdce23700 0 -- 172.21.15.107:0/1663484006 >> v1:172.21.15.107:6808/33988 conn(0x55598308c0e0 legacy=0x7febc803b250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until injecting socket failure 2021-02-12T17:59:22.629 INFO:teuthology.orchestra.run.smithi107.stdout:Total time run: 30.0062 2021-02-12T17:59:22.629 INFO:teuthology.orchestra.run.smithi107.stdout:Total writes made: 31133 2021-02-12T17:59:22.629 INFO:teuthology.orchestra.run.smithi107.stdout:Write size: 4096 2021-02-12T17:59:22.629 INFO:teuthology.orchestra.run.smithi107.stdout:Object size: 4096 2021-02-12T17:59:22.629 INFO:teuthology.orchestra.run.smithi107.stdout:Bandwidth (MB/sec): 4.05293 2021-02-12T17:59:22.629 INFO:teuthology.orchestra.run.smithi107.stdout:Stddev Bandwidth: 1.20424 2021-02-12T17:59:22.629 INFO:teuthology.orchestra.run.smithi107.stdout:Max bandwidth (MB/sec): 6.16406 2021-02-12T17:59:22.629 INFO:teuthology.orchestra.run.smithi107.stdout:Min bandwidth (MB/sec): 0.628906 2021-02-12T17:59:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:Average IOPS: 1037 2021-02-12T17:59:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:Stddev IOPS: 308.285 2021-02-12T17:59:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:Max IOPS: 1578 2021-02-12T17:59:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:Min IOPS: 161 2021-02-12T17:59:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:Average Latency(s): 0.0154166 2021-02-12T17:59:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:Stddev Latency(s): 0.0334145 2021-02-12T17:59:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:Max latency(s): 1.41286 2021-02-12T17:59:22.630 INFO:teuthology.orchestra.run.smithi107.stdout:Min latency(s): 0.00394581 2021-02-12T17:59:22.719 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2021-02-12T17:59:22.843 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:0 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:1 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:2 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator fragmentation block 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator score block 2021-02-12T17:59:22.844 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2021-02-12T17:59:22.845 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-12T17:59:22.845 INFO:tasks.osd_backfill.ceph_manager:waiting on admin_socket for osd-1, ['dump_ops_in_flight'] 2021-02-12T17:59:22.879 INFO:tasks.ceph.osd.1.smithi107.stderr:2021-02-12T17:59:22.878+0000 7f64bc5b9080 -1 osd.1 25 log_to_monitors {default=true} 2021-02-12T17:59:27.846 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.1.asok dump_ops_in_flight 2021-02-12T17:59:27.986 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2021-02-12T17:59:27.986 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2021-02-12T17:59:27.986 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2021-02-12T17:59:27.986 INFO:teuthology.orchestra.run.smithi107.stdout:} 2021-02-12T17:59:27.997 INFO:tasks.osd_backfill:err is 0 2021-02-12T17:59:27.997 INFO:tasks.osd_backfill.ceph_manager:waiting for osd.1 to be up 2021-02-12T17:59:27.997 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-02-12T17:59:28.325 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:28.325 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":30,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T17:59:25.651606+0000","last_up_change":"2021-02-12T17:59:23.641264+0000","last_in_change":"2021-02-12T17:57:50.561433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-12T17:57:23.210299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":13,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":33986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":33986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":33986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":33986}]},"public_addr":"172.21.15.107:6802/33986","cluster_addr":"172.21.15.107:6804/33986","heartbeat_back_addr":"172.21.15.107:6807/33986","heartbeat_front_addr":"172.21.15.107:6806/33986","state":["exists"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":28,"up_thru":28,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":39656}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":39656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":39656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":39656}]},"public_addr":"172.21.15.107:6800/39656","cluster_addr":"172.21.15.107:6801/39656","heartbeat_back_addr":"172.21.15.107:6803/39656","heartbeat_front_addr":"172.21.15.107:6802/39656","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":65536,"last_purged_snaps_scrub":"2021-02-12T17:57:20.700614+0000","dead_epoch":13},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.801932+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.752083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[2]},{"pgid":"1.1","osds":[2]},{"pgid":"1.2","osds":[2]},{"pgid":"1.3","osds":[2]},{"pgid":"1.4","osds":[2]},{"pgid":"1.5","osds":[2]},{"pgid":"1.6","osds":[2]},{"pgid":"1.7","osds":[2]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T17:59:28.338 INFO:tasks.osd_backfill.ceph_manager:osd.1 is up 2021-02-12T17:59:28.339 INFO:tasks.osd_backfill.ceph_manager:waiting for osd.2 to be up 2021-02-12T17:59:28.339 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-02-12T17:59:28.698 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:28.699 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":30,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T17:59:25.651606+0000","last_up_change":"2021-02-12T17:59:23.641264+0000","last_in_change":"2021-02-12T17:57:50.561433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-12T17:57:23.210299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":13,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":33986}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":33986}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":33986}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":33986}]},"public_addr":"172.21.15.107:6802/33986","cluster_addr":"172.21.15.107:6804/33986","heartbeat_back_addr":"172.21.15.107:6807/33986","heartbeat_front_addr":"172.21.15.107:6806/33986","state":["exists"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":28,"up_thru":28,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":39656}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":39656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":39656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":39656}]},"public_addr":"172.21.15.107:6800/39656","cluster_addr":"172.21.15.107:6801/39656","heartbeat_back_addr":"172.21.15.107:6803/39656","heartbeat_front_addr":"172.21.15.107:6802/39656","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":29,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":65536,"last_purged_snaps_scrub":"2021-02-12T17:57:20.700614+0000","dead_epoch":13},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.801932+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.752083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[{"pgid":"1.0","osds":[2]},{"pgid":"1.1","osds":[2]},{"pgid":"1.2","osds":[2]},{"pgid":"1.3","osds":[2]},{"pgid":"1.4","osds":[2]},{"pgid":"1.5","osds":[2]},{"pgid":"1.6","osds":[2]},{"pgid":"1.7","osds":[2]}],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T17:59:28.713 INFO:tasks.osd_backfill.ceph_manager:osd.2 is up 2021-02-12T17:59:28.713 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-02-12T17:59:28.890 INFO:teuthology.orchestra.run.smithi107.stdout:120259084291 2021-02-12T17:59:28.891 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-02-12T17:59:29.061 INFO:teuthology.orchestra.run.smithi107.stdout:34359738398 2021-02-12T17:59:29.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-12T17:59:29.415 INFO:teuthology.orchestra.run.smithi107.stdout:120259084290 2021-02-12T17:59:29.430 INFO:tasks.osd_backfill.ceph_manager:need seq 120259084291 got 120259084290 for osd.1 2021-02-12T17:59:30.431 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-12T17:59:30.773 INFO:teuthology.orchestra.run.smithi107.stdout:120259084291 2021-02-12T17:59:30.787 INFO:tasks.osd_backfill.ceph_manager:need seq 120259084291 got 120259084291 for osd.1 2021-02-12T17:59:30.787 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-12T17:59:31.129 INFO:teuthology.orchestra.run.smithi107.stdout:34359738398 2021-02-12T17:59:31.143 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738398 got 34359738398 for osd.2 2021-02-12T17:59:31.143 INFO:tasks.osd_backfill.ceph_manager:waiting for recovery to complete 2021-02-12T17:59:31.143 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:31.477 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:31.479 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:31.490 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2021-02-12T17:59:30.419656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17456,"num_bytes_recovered":71495704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":494104,"kb_used_data":365832,"kb_used_omap":0,"kb_used_meta":128255,"kb_avail":188249576,"statfs":{"total":193273528320,"available":192767565824,"internally_reserved":0,"allocated":374611968,"data_stored":374235736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":131333172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-37762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":2265088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.776274"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6761","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941081+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:25.941081+0000","last_peered":"2021-02-12T17:59:25.941081+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:25.941081+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12494","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.908055+0000","last_change":"2021-02-12T17:59:25.909186+0000","last_active":"2021-02-12T17:59:25.908055+0000","last_peered":"2021-02-12T17:59:25.908055+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:25.908055+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2834,"num_bytes_recovered":11608064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12583","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.946989+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:25.946989+0000","last_peered":"2021-02-12T17:59:25.946989+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:25.946989+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12278","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.928237+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:25.928237+0000","last_peered":"2021-02-12T17:59:25.928237+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:25.928237+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12143","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.948295+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:25.948295+0000","last_peered":"2021-02-12T17:59:25.948295+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:25.948295+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6725","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941216+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:25.941216+0000","last_peered":"2021-02-12T17:59:25.941216+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:25.941216+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"13509","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:29.045903+0000","last_change":"2021-02-12T17:59:25.905420+0000","last_active":"2021-02-12T17:59:29.045903+0000","last_peered":"2021-02-12T17:59:29.045903+0000","last_clean":"2021-02-12T17:59:10.111484+0000","last_became_active":"2021-02-12T17:59:25.888275+0000","last_became_peered":"2021-02-12T17:59:25.888275+0000","last_unstale":"2021-02-12T17:59:29.045903+0000","last_undegraded":"2021-02-12T17:59:25.700475+0000","last_fullsized":"2021-02-12T17:59:25.662455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3384,"num_bytes_recovered":13860864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5170}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12579","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.935560+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:25.935560+0000","last_peered":"2021-02-12T17:59:25.935560+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:25.935560+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17456,"num_bytes_recovered":71495704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":441761792,"data_stored":441753648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":215292,"kb_used_omap":0,"kb_used_meta":87488,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":220459008,"data_stored":220270892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89587712},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.56299999999999994}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191316,"kb_used_data":150540,"kb_used_omap":0,"kb_used_meta":40767,"kb_avail":94180524,"statfs":{"total":96636764160,"available":96440856576,"internally_reserved":0,"allocated":154152960,"data_stored":153964844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":41745460},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}]}} 2021-02-12T17:59:31.491 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:31.847 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:31.849 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:31.860 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2021-02-12T17:59:30.419656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17456,"num_bytes_recovered":71495704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":494104,"kb_used_data":365832,"kb_used_omap":0,"kb_used_meta":128255,"kb_avail":188249576,"statfs":{"total":193273528320,"available":192767565824,"internally_reserved":0,"allocated":374611968,"data_stored":374235736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":131333172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-37762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":2265088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.776274"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6761","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941081+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:25.941081+0000","last_peered":"2021-02-12T17:59:25.941081+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:25.941081+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12494","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.908055+0000","last_change":"2021-02-12T17:59:25.909186+0000","last_active":"2021-02-12T17:59:25.908055+0000","last_peered":"2021-02-12T17:59:25.908055+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:25.908055+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2834,"num_bytes_recovered":11608064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12583","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.946989+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:25.946989+0000","last_peered":"2021-02-12T17:59:25.946989+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:25.946989+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12278","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.928237+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:25.928237+0000","last_peered":"2021-02-12T17:59:25.928237+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:25.928237+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12143","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.948295+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:25.948295+0000","last_peered":"2021-02-12T17:59:25.948295+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:25.948295+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6725","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941216+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:25.941216+0000","last_peered":"2021-02-12T17:59:25.941216+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:25.941216+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"13509","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:29.045903+0000","last_change":"2021-02-12T17:59:25.905420+0000","last_active":"2021-02-12T17:59:29.045903+0000","last_peered":"2021-02-12T17:59:29.045903+0000","last_clean":"2021-02-12T17:59:10.111484+0000","last_became_active":"2021-02-12T17:59:25.888275+0000","last_became_peered":"2021-02-12T17:59:25.888275+0000","last_unstale":"2021-02-12T17:59:29.045903+0000","last_undegraded":"2021-02-12T17:59:25.700475+0000","last_fullsized":"2021-02-12T17:59:25.662455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3384,"num_bytes_recovered":13860864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5170}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12579","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.935560+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:25.935560+0000","last_peered":"2021-02-12T17:59:25.935560+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:25.935560+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17456,"num_bytes_recovered":71495704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":441761792,"data_stored":441753648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":215292,"kb_used_omap":0,"kb_used_meta":87488,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":220459008,"data_stored":220270892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89587712},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.56299999999999994}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191316,"kb_used_data":150540,"kb_used_omap":0,"kb_used_meta":40767,"kb_avail":94180524,"statfs":{"total":96636764160,"available":96440856576,"internally_reserved":0,"allocated":154152960,"data_stored":153964844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":41745460},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}]}} 2021-02-12T17:59:31.861 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:32.228 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:32.231 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:32.242 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":92,"stamp":"2021-02-12T17:59:30.419656+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17456,"num_bytes_recovered":71495704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":494104,"kb_used_data":365832,"kb_used_omap":0,"kb_used_meta":128255,"kb_avail":188249576,"statfs":{"total":193273528320,"available":192767565824,"internally_reserved":0,"allocated":374611968,"data_stored":374235736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":131333172},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-37762,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":553,"num_bytes_recovered":2265088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"6.776274"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6761","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941081+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:25.941081+0000","last_peered":"2021-02-12T17:59:25.941081+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:25.941081+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12494","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.908055+0000","last_change":"2021-02-12T17:59:25.909186+0000","last_active":"2021-02-12T17:59:25.908055+0000","last_peered":"2021-02-12T17:59:25.908055+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:25.908055+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2834,"num_bytes_recovered":11608064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12583","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.946989+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:25.946989+0000","last_peered":"2021-02-12T17:59:25.946989+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:25.946989+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12278","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.928237+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:25.928237+0000","last_peered":"2021-02-12T17:59:25.928237+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:25.928237+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12143","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.948295+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:25.948295+0000","last_peered":"2021-02-12T17:59:25.948295+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:25.948295+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6725","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941216+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:25.941216+0000","last_peered":"2021-02-12T17:59:25.941216+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:25.941216+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"13509","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:29.045903+0000","last_change":"2021-02-12T17:59:25.905420+0000","last_active":"2021-02-12T17:59:29.045903+0000","last_peered":"2021-02-12T17:59:29.045903+0000","last_clean":"2021-02-12T17:59:10.111484+0000","last_became_active":"2021-02-12T17:59:25.888275+0000","last_became_peered":"2021-02-12T17:59:25.888275+0000","last_unstale":"2021-02-12T17:59:29.045903+0000","last_undegraded":"2021-02-12T17:59:25.700475+0000","last_fullsized":"2021-02-12T17:59:25.662455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3384,"num_bytes_recovered":13860864,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5170}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12579","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.935560+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:25.935560+0000","last_peered":"2021-02-12T17:59:25.935560+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:25.935560+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15999,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":17456,"num_bytes_recovered":71495704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":441761792,"data_stored":441753648,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738398,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":215292,"kb_used_omap":0,"kb_used_meta":87488,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":220459008,"data_stored":220270892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89587712},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.56299999999999994}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084291,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":191316,"kb_used_data":150540,"kb_used_omap":0,"kb_used_meta":40767,"kb_avail":94180524,"statfs":{"total":96636764160,"available":96440856576,"internally_reserved":0,"allocated":154152960,"data_stored":153964844,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":41745460},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.371}]}]}]}} 2021-02-12T17:59:35.243 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:35.927 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:35.929 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:35.942 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2021-02-12T17:59:35.695962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18435,"num_bytes_recovered":75505688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":502812,"kb_used_data":369100,"kb_used_omap":0,"kb_used_meta":133695,"kb_avail":188240868,"statfs":{"total":193273528320,"available":192758648832,"internally_reserved":0,"allocated":377958400,"data_stored":377582168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":136903732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-38741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1532,"num_bytes_recovered":6275072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.039994"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6761","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941081+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:25.941081+0000","last_peered":"2021-02-12T17:59:25.941081+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:25.941081+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12494","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.908055+0000","last_change":"2021-02-12T17:59:25.909186+0000","last_active":"2021-02-12T17:59:25.908055+0000","last_peered":"2021-02-12T17:59:25.908055+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:25.908055+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2834,"num_bytes_recovered":11608064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12583","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.946989+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:25.946989+0000","last_peered":"2021-02-12T17:59:25.946989+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:25.946989+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12278","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.928237+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:25.928237+0000","last_peered":"2021-02-12T17:59:25.928237+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:25.928237+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12143","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.948295+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:25.948295+0000","last_peered":"2021-02-12T17:59:25.948295+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:25.948295+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6725","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941216+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:25.941216+0000","last_peered":"2021-02-12T17:59:25.941216+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:25.941216+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"15467","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:33.471137+0000","last_change":"2021-02-12T17:59:25.905420+0000","last_active":"2021-02-12T17:59:33.471137+0000","last_peered":"2021-02-12T17:59:33.471137+0000","last_clean":"2021-02-12T17:59:10.111484+0000","last_became_active":"2021-02-12T17:59:25.888275+0000","last_became_peered":"2021-02-12T17:59:25.888275+0000","last_unstale":"2021-02-12T17:59:33.471137+0000","last_undegraded":"2021-02-12T17:59:25.700475+0000","last_fullsized":"2021-02-12T17:59:25.662455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4363,"num_bytes_recovered":17870848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12579","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.935560+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:25.935560+0000","last_peered":"2021-02-12T17:59:25.935560+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:25.935560+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18435,"num_bytes_recovered":75505688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445411328,"data_stored":445403184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":215292,"kb_used_omap":0,"kb_used_meta":87488,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":220459008,"data_stored":220270892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89587712},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42299999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200024,"kb_used_data":153808,"kb_used_omap":0,"kb_used_meta":46207,"kb_avail":94171816,"statfs":{"total":96636764160,"available":96431939584,"internally_reserved":0,"allocated":157499392,"data_stored":157311276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":47316020},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]}]}} 2021-02-12T17:59:35.943 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:36.288 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:36.290 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:36.303 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":95,"stamp":"2021-02-12T17:59:35.695962+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18435,"num_bytes_recovered":75505688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":502812,"kb_used_data":369100,"kb_used_omap":0,"kb_used_meta":133695,"kb_avail":188240868,"statfs":{"total":193273528320,"available":192758648832,"internally_reserved":0,"allocated":377958400,"data_stored":377582168,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":136903732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-38741,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1532,"num_bytes_recovered":6275072,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.039994"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6761","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941081+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:25.941081+0000","last_peered":"2021-02-12T17:59:25.941081+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:25.941081+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12494","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.908055+0000","last_change":"2021-02-12T17:59:25.909186+0000","last_active":"2021-02-12T17:59:25.908055+0000","last_peered":"2021-02-12T17:59:25.908055+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:25.908055+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2834,"num_bytes_recovered":11608064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12583","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.946989+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:25.946989+0000","last_peered":"2021-02-12T17:59:25.946989+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:25.946989+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12278","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.928237+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:25.928237+0000","last_peered":"2021-02-12T17:59:25.928237+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:25.928237+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12143","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.948295+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:25.948295+0000","last_peered":"2021-02-12T17:59:25.948295+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:25.948295+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6725","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941216+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:25.941216+0000","last_peered":"2021-02-12T17:59:25.941216+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:25.941216+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"15467","reported_epoch":"30","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:33.471137+0000","last_change":"2021-02-12T17:59:25.905420+0000","last_active":"2021-02-12T17:59:33.471137+0000","last_peered":"2021-02-12T17:59:33.471137+0000","last_clean":"2021-02-12T17:59:10.111484+0000","last_became_active":"2021-02-12T17:59:25.888275+0000","last_became_peered":"2021-02-12T17:59:25.888275+0000","last_unstale":"2021-02-12T17:59:33.471137+0000","last_undegraded":"2021-02-12T17:59:25.700475+0000","last_fullsized":"2021-02-12T17:59:25.662455+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":23,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":539,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4363,"num_bytes_recovered":17870848,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6149}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12579","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.935560+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:25.935560+0000","last_peered":"2021-02-12T17:59:25.935560+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:25.935560+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":15020,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18435,"num_bytes_recovered":75505688,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":445411328,"data_stored":445403184,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":8,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":215292,"kb_used_omap":0,"kb_used_meta":87488,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":220459008,"data_stored":220270892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89587712},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42299999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084292,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":200024,"kb_used_data":153808,"kb_used_omap":0,"kb_used_meta":46207,"kb_avail":94171816,"statfs":{"total":96636764160,"available":96431939584,"internally_reserved":0,"allocated":157499392,"data_stored":157311276,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":47316020},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36199999999999999}]}]}]}} 2021-02-12T17:59:39.305 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:39.633 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:39.635 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:39.646 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2021-02-12T17:59:38.420995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18974,"num_bytes_recovered":77713432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":518644,"kb_used_data":374500,"kb_used_omap":0,"kb_used_meta":144127,"kb_avail":188225036,"statfs":{"total":193273528320,"available":192742436864,"internally_reserved":0,"allocated":383488000,"data_stored":383100481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":147586100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":6217728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001339"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6761","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941081+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:25.941081+0000","last_peered":"2021-02-12T17:59:25.941081+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:25.941081+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12494","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.908055+0000","last_change":"2021-02-12T17:59:25.909186+0000","last_active":"2021-02-12T17:59:25.908055+0000","last_peered":"2021-02-12T17:59:25.908055+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:25.908055+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2834,"num_bytes_recovered":11608064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12583","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.946989+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:25.946989+0000","last_peered":"2021-02-12T17:59:25.946989+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:25.946989+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12278","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.928237+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:25.928237+0000","last_peered":"2021-02-12T17:59:25.928237+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:25.928237+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12143","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.948295+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:25.948295+0000","last_peered":"2021-02-12T17:59:25.948295+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:25.948295+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6725","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941216+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:25.941216+0000","last_peered":"2021-02-12T17:59:25.941216+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:25.941216+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12408","reported_epoch":"32","state":"active+clean","last_fresh":"2021-02-12T17:59:36.712322+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:36.712322+0000","last_peered":"2021-02-12T17:59:36.712322+0000","last_clean":"2021-02-12T17:59:36.712322+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:36.712322+0000","last_undegraded":"2021-02-12T17:59:36.712322+0000","last_fullsized":"2021-02-12T17:59:36.712322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12579","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.935560+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:25.935560+0000","last_peered":"2021-02-12T17:59:25.935560+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:25.935560+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18974,"num_bytes_recovered":77713432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450707456,"data_stored":450699312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":215292,"kb_used_omap":0,"kb_used_meta":87488,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":220459008,"data_stored":220270892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89587712},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42299999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":215856,"kb_used_data":159208,"kb_used_omap":0,"kb_used_meta":56639,"kb_avail":94155984,"statfs":{"total":96636764160,"available":96415727616,"internally_reserved":0,"allocated":163028992,"data_stored":162829589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":57998388},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]}]}]}} 2021-02-12T17:59:39.647 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:39.981 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:39.983 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:39.994 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":98,"stamp":"2021-02-12T17:59:38.420995+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18974,"num_bytes_recovered":77713432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":518644,"kb_used_data":374500,"kb_used_omap":0,"kb_used_meta":144127,"kb_avail":188225036,"statfs":{"total":193273528320,"available":192742436864,"internally_reserved":0,"allocated":383488000,"data_stored":383100481,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":147586100},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1518,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1518,"num_bytes_recovered":6217728,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001339"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6761","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941081+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:25.941081+0000","last_peered":"2021-02-12T17:59:25.941081+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:25.941081+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12494","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.908055+0000","last_change":"2021-02-12T17:59:25.909186+0000","last_active":"2021-02-12T17:59:25.908055+0000","last_peered":"2021-02-12T17:59:25.908055+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:25.908055+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2094,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2834,"num_bytes_recovered":11608064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4678}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12583","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.946989+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:25.946989+0000","last_peered":"2021-02-12T17:59:25.946989+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:25.946989+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12278","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.928237+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:25.928237+0000","last_peered":"2021-02-12T17:59:25.928237+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:25.928237+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12143","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.948295+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:25.948295+0000","last_peered":"2021-02-12T17:59:25.948295+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:25.948295+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6725","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.941216+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:25.941216+0000","last_peered":"2021-02-12T17:59:25.941216+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:25.941216+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12408","reported_epoch":"32","state":"active+clean","last_fresh":"2021-02-12T17:59:36.712322+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:36.712322+0000","last_peered":"2021-02-12T17:59:36.712322+0000","last_clean":"2021-02-12T17:59:36.712322+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:36.712322+0000","last_undegraded":"2021-02-12T17:59:36.712322+0000","last_fullsized":"2021-02-12T17:59:36.712322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12579","reported_epoch":"30","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:25.935560+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:25.935560+0000","last_peered":"2021-02-12T17:59:25.935560+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:25.935560+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":14481,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":18974,"num_bytes_recovered":77713432,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450707456,"data_stored":450699312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738399,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302788,"kb_used_data":215292,"kb_used_omap":0,"kb_used_meta":87488,"kb_avail":94069052,"statfs":{"total":96636764160,"available":96326709248,"internally_reserved":0,"allocated":220459008,"data_stored":220270892,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89587712},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.42299999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":215856,"kb_used_data":159208,"kb_used_omap":0,"kb_used_meta":56639,"kb_avail":94155984,"statfs":{"total":96636764160,"available":96415727616,"internally_reserved":0,"allocated":163028992,"data_stored":162829589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":57998388},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]}]}]}} 2021-02-12T17:59:42.996 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:43.331 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:43.333 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:43.344 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2021-02-12T17:59:42.421599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19736,"num_bytes_recovered":80834584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":518724,"kb_used_data":374516,"kb_used_omap":0,"kb_used_meta":144191,"kb_avail":188224956,"statfs":{"total":193273528320,"available":192742354944,"internally_reserved":0,"allocated":383504384,"data_stored":383105578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":147651636},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1301,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001290"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6763","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702981+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:35.702981+0000","last_peered":"2021-02-12T17:59:35.702981+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:35.702981+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"14021","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:38.470908+0000","last_change":"2021-02-12T17:59:35.008679+0000","last_active":"2021-02-12T17:59:38.470908+0000","last_peered":"2021-02-12T17:59:38.470908+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:38.470908+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":14729216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5440}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12585","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702998+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:35.702998+0000","last_peered":"2021-02-12T17:59:35.702998+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:35.702998+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12280","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.704778+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:35.704778+0000","last_peered":"2021-02-12T17:59:35.704778+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:35.704778+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12145","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702825+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:35.702825+0000","last_peered":"2021-02-12T17:59:35.702825+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:35.702825+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6727","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702965+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:35.702965+0000","last_peered":"2021-02-12T17:59:35.702965+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:35.702965+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12408","reported_epoch":"32","state":"active+clean","last_fresh":"2021-02-12T17:59:36.712322+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:36.712322+0000","last_peered":"2021-02-12T17:59:36.712322+0000","last_clean":"2021-02-12T17:59:36.712322+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:36.712322+0000","last_undegraded":"2021-02-12T17:59:36.712322+0000","last_fullsized":"2021-02-12T17:59:36.712322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12581","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702971+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:35.702971+0000","last_peered":"2021-02-12T17:59:35.702971+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:35.702971+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19736,"num_bytes_recovered":80834584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450707456,"data_stored":450699312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738400,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302868,"kb_used_data":215308,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068972,"statfs":{"total":96636764160,"available":96326627328,"internally_reserved":0,"allocated":220475392,"data_stored":220275989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.38}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":215856,"kb_used_data":159208,"kb_used_omap":0,"kb_used_meta":56639,"kb_avail":94155984,"statfs":{"total":96636764160,"available":96415727616,"internally_reserved":0,"allocated":163028992,"data_stored":162829589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":57998388},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]}]}]}} 2021-02-12T17:59:43.345 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:43.681 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:43.684 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:43.695 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":100,"stamp":"2021-02-12T17:59:42.421599+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19736,"num_bytes_recovered":80834584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":518724,"kb_used_data":374516,"kb_used_omap":0,"kb_used_meta":144191,"kb_avail":188224956,"statfs":{"total":193273528320,"available":192742354944,"internally_reserved":0,"allocated":383504384,"data_stored":383105578,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":147651636},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1301,"num_bytes_recovered":5328896,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001290"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6763","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702981+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:35.702981+0000","last_peered":"2021-02-12T17:59:35.702981+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:35.702981+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"14021","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:38.470908+0000","last_change":"2021-02-12T17:59:35.008679+0000","last_active":"2021-02-12T17:59:38.470908+0000","last_peered":"2021-02-12T17:59:38.470908+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:38.470908+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1332,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3596,"num_bytes_recovered":14729216,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5440}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12585","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702998+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:35.702998+0000","last_peered":"2021-02-12T17:59:35.702998+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:35.702998+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12280","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.704778+0000","last_change":"2021-02-12T17:59:25.929429+0000","last_active":"2021-02-12T17:59:35.704778+0000","last_peered":"2021-02-12T17:59:35.704778+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:35.704778+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2102,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2746,"num_bytes_recovered":11247616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4631}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12145","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702825+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:35.702825+0000","last_peered":"2021-02-12T17:59:35.702825+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:35.702825+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6727","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702965+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:35.702965+0000","last_peered":"2021-02-12T17:59:35.702965+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:35.702965+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12408","reported_epoch":"32","state":"active+clean","last_fresh":"2021-02-12T17:59:36.712322+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:36.712322+0000","last_peered":"2021-02-12T17:59:36.712322+0000","last_clean":"2021-02-12T17:59:36.712322+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:36.712322+0000","last_undegraded":"2021-02-12T17:59:36.712322+0000","last_fullsized":"2021-02-12T17:59:36.712322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12581","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702971+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:35.702971+0000","last_peered":"2021-02-12T17:59:35.702971+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:35.702971+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":13719,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":19736,"num_bytes_recovered":80834584,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":450707456,"data_stored":450699312,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738400,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302868,"kb_used_data":215308,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068972,"statfs":{"total":96636764160,"available":96326627328,"internally_reserved":0,"allocated":220475392,"data_stored":220275989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.38}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084293,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":215856,"kb_used_data":159208,"kb_used_omap":0,"kb_used_meta":56639,"kb_avail":94155984,"statfs":{"total":96636764160,"available":96415727616,"internally_reserved":0,"allocated":163028992,"data_stored":162829589,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":57998388},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.30299999999999999}]}]}]}} 2021-02-12T17:59:46.697 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:47.037 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:47.039 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:47.050 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2021-02-12T17:59:46.422192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21097,"num_bytes_recovered":86409240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":529500,"kb_used_data":379276,"kb_used_omap":0,"kb_used_meta":150207,"kb_avail":188214180,"statfs":{"total":193273528320,"available":192731320320,"internally_reserved":0,"allocated":388378624,"data_stored":387979818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":153812020},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":8695808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001196"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6763","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702981+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:35.702981+0000","last_peered":"2021-02-12T17:59:35.702981+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:35.702981+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"16686","reported_epoch":"32","state":"active+undersized+remapped","last_fresh":"2021-02-12T17:59:43.271930+0000","last_change":"2021-02-12T17:59:43.272124+0000","last_active":"2021-02-12T17:59:43.271930+0000","last_peered":"2021-02-12T17:59:43.271930+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:43.271930+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"2","objects":6772}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12585","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702998+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:35.702998+0000","last_peered":"2021-02-12T17:59:35.702998+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:35.702998+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12339","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.470357+0000","last_change":"2021-02-12T17:59:43.272672+0000","last_active":"2021-02-12T17:59:43.470357+0000","last_peered":"2021-02-12T17:59:43.470357+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:43.470357+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2775,"num_bytes_recovered":11366400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4660}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12145","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702825+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:35.702825+0000","last_peered":"2021-02-12T17:59:35.702825+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:35.702825+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6727","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702965+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:35.702965+0000","last_peered":"2021-02-12T17:59:35.702965+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:35.702965+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12408","reported_epoch":"32","state":"active+clean","last_fresh":"2021-02-12T17:59:36.712322+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:36.712322+0000","last_peered":"2021-02-12T17:59:36.712322+0000","last_clean":"2021-02-12T17:59:36.712322+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:36.712322+0000","last_undegraded":"2021-02-12T17:59:36.712322+0000","last_fullsized":"2021-02-12T17:59:36.712322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12581","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702971+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:35.702971+0000","last_peered":"2021-02-12T17:59:35.702971+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:35.702971+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21097,"num_bytes_recovered":86409240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":456122368,"data_stored":456114224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302868,"kb_used_data":215308,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068972,"statfs":{"total":96636764160,"available":96326627328,"internally_reserved":0,"allocated":220475392,"data_stored":220275989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.35299999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":226632,"kb_used_data":163968,"kb_used_omap":0,"kb_used_meta":62655,"kb_avail":94145208,"statfs":{"total":96636764160,"available":96404692992,"internally_reserved":0,"allocated":167903232,"data_stored":167703829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":64158772},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]}]}]}} 2021-02-12T17:59:47.051 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:47.379 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:47.382 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:47.393 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":104,"stamp":"2021-02-12T17:59:46.422192+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21097,"num_bytes_recovered":86409240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":529500,"kb_used_data":379276,"kb_used_omap":0,"kb_used_meta":150207,"kb_avail":188214180,"statfs":{"total":193273528320,"available":192731320320,"internally_reserved":0,"allocated":388378624,"data_stored":387979818,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":153812020},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2123,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2123,"num_bytes_recovered":8695808,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001196"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6763","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702981+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:35.702981+0000","last_peered":"2021-02-12T17:59:35.702981+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:35.702981+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"16686","reported_epoch":"32","state":"active+undersized+remapped","last_fresh":"2021-02-12T17:59:43.271930+0000","last_change":"2021-02-12T17:59:43.272124+0000","last_active":"2021-02-12T17:59:43.271930+0000","last_peered":"2021-02-12T17:59:43.271930+0000","last_clean":"2021-02-12T17:59:10.093371+0000","last_became_active":"2021-02-12T17:59:25.902072+0000","last_became_peered":"2021-02-12T17:59:25.902072+0000","last_unstale":"2021-02-12T17:59:43.271930+0000","last_undegraded":"2021-02-12T17:59:25.700358+0000","last_fullsized":"2021-02-12T17:59:25.662804+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":25,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2","1"],"object_location_counts":[{"shards":"2","objects":6772}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12585","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702998+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:35.702998+0000","last_peered":"2021-02-12T17:59:35.702998+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:35.702998+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12339","reported_epoch":"32","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.470357+0000","last_change":"2021-02-12T17:59:43.272672+0000","last_active":"2021-02-12T17:59:43.470357+0000","last_peered":"2021-02-12T17:59:43.470357+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:43.470357+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2073,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2775,"num_bytes_recovered":11366400,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4660}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12145","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702825+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:35.702825+0000","last_peered":"2021-02-12T17:59:35.702825+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:35.702825+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6727","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702965+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:35.702965+0000","last_peered":"2021-02-12T17:59:35.702965+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:35.702965+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12408","reported_epoch":"32","state":"active+clean","last_fresh":"2021-02-12T17:59:36.712322+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:36.712322+0000","last_peered":"2021-02-12T17:59:36.712322+0000","last_clean":"2021-02-12T17:59:36.712322+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:36.712322+0000","last_undegraded":"2021-02-12T17:59:36.712322+0000","last_fullsized":"2021-02-12T17:59:36.712322+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12581","reported_epoch":"31","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:35.702971+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:35.702971+0000","last_peered":"2021-02-12T17:59:35.702971+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:35.702971+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":12358,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":21097,"num_bytes_recovered":86409240,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":456122368,"data_stored":456114224,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":9,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302868,"kb_used_data":215308,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068972,"statfs":{"total":96636764160,"available":96326627328,"internally_reserved":0,"allocated":220475392,"data_stored":220275989,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.33100000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.35299999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084294,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":226632,"kb_used_data":163968,"kb_used_omap":0,"kb_used_meta":62655,"kb_avail":94145208,"statfs":{"total":96636764160,"available":96404692992,"internally_reserved":0,"allocated":167903232,"data_stored":167703829,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":64158772},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.37}]}]}]}} 2021-02-12T17:59:50.395 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:50.744 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:50.746 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:50.758 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2021-02-12T17:59:50.422812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":91070488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":524432,"kb_used_data":384384,"kb_used_omap":0,"kb_used_meta":140031,"kb_avail":188219248,"statfs":{"total":193273528320,"available":192736509952,"internally_reserved":0,"allocated":393609216,"data_stored":393187736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":143391796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2499,"num_bytes_recovered":10235904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001212"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6765","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.729838+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:43.729838+0000","last_peered":"2021-02-12T17:59:43.729838+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:43.729838+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12500","reported_epoch":"34","state":"active+clean","last_fresh":"2021-02-12T17:59:44.742387+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:44.742387+0000","last_peered":"2021-02-12T17:59:44.742387+0000","last_clean":"2021-02-12T17:59:44.742387+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:44.742387+0000","last_undegraded":"2021-02-12T17:59:44.742387+0000","last_fullsized":"2021-02-12T17:59:44.742387+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12587","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.726813+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:43.726813+0000","last_peered":"2021-02-12T17:59:43.726813+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:43.726813+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"14617","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:48.473023+0000","last_change":"2021-02-12T17:59:43.272672+0000","last_active":"2021-02-12T17:59:48.473023+0000","last_peered":"2021-02-12T17:59:48.473023+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:48.473023+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3913,"num_bytes_recovered":16027648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12147","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.727020+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:43.727020+0000","last_peered":"2021-02-12T17:59:43.727020+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:43.727020+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6729","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.726653+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:43.726653+0000","last_peered":"2021-02-12T17:59:43.726653+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:43.726653+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12410","reported_epoch":"33","state":"active+clean","last_fresh":"2021-02-12T17:59:43.722385+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:43.722385+0000","last_peered":"2021-02-12T17:59:43.722385+0000","last_clean":"2021-02-12T17:59:43.722385+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:43.722385+0000","last_undegraded":"2021-02-12T17:59:43.722385+0000","last_fullsized":"2021-02-12T17:59:43.722385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12583","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.728485+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:43.728485+0000","last_peered":"2021-02-12T17:59:43.728485+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:43.728485+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":91070488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":460853248,"data_stored":460845104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302884,"kb_used_data":215324,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068956,"statfs":{"total":96636764160,"available":96326610944,"internally_reserved":0,"allocated":220491776,"data_stored":220281036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.39200000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221548,"kb_used_data":169060,"kb_used_omap":0,"kb_used_meta":52479,"kb_avail":94150292,"statfs":{"total":96636764160,"available":96409899008,"internally_reserved":0,"allocated":173117440,"data_stored":172906700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":53738548},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]}]}} 2021-02-12T17:59:50.759 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:51.095 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:51.097 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:51.108 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":106,"stamp":"2021-02-12T17:59:50.422812+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":91070488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":10,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":524432,"kb_used_data":384384,"kb_used_omap":0,"kb_used_meta":140031,"kb_avail":188219248,"statfs":{"total":193273528320,"available":192736509952,"internally_reserved":0,"allocated":393609216,"data_stored":393187736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":143391796},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2499,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2499,"num_bytes_recovered":10235904,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001212"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6765","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.729838+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:43.729838+0000","last_peered":"2021-02-12T17:59:43.729838+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:43.729838+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12500","reported_epoch":"34","state":"active+clean","last_fresh":"2021-02-12T17:59:44.742387+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:44.742387+0000","last_peered":"2021-02-12T17:59:44.742387+0000","last_clean":"2021-02-12T17:59:44.742387+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:44.742387+0000","last_undegraded":"2021-02-12T17:59:44.742387+0000","last_fullsized":"2021-02-12T17:59:44.742387+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12587","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.726813+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:43.726813+0000","last_peered":"2021-02-12T17:59:43.726813+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:43.726813+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"14617","reported_epoch":"34","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:48.473023+0000","last_change":"2021-02-12T17:59:43.272672+0000","last_active":"2021-02-12T17:59:48.473023+0000","last_peered":"2021-02-12T17:59:48.473023+0000","last_clean":"2021-02-12T17:59:10.102358+0000","last_became_active":"2021-02-12T17:59:25.921283+0000","last_became_peered":"2021-02-12T17:59:25.921283+0000","last_unstale":"2021-02-12T17:59:48.473023+0000","last_undegraded":"2021-02-12T17:59:25.701345+0000","last_fullsized":"2021-02-12T17:59:25.662788+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":935,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3913,"num_bytes_recovered":16027648,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5798}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12147","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.727020+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:43.727020+0000","last_peered":"2021-02-12T17:59:43.727020+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:43.727020+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6729","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.726653+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:43.726653+0000","last_peered":"2021-02-12T17:59:43.726653+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:43.726653+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12410","reported_epoch":"33","state":"active+clean","last_fresh":"2021-02-12T17:59:43.722385+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:43.722385+0000","last_peered":"2021-02-12T17:59:43.722385+0000","last_clean":"2021-02-12T17:59:43.722385+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:43.722385+0000","last_undegraded":"2021-02-12T17:59:43.722385+0000","last_fullsized":"2021-02-12T17:59:43.722385+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"12583","reported_epoch":"33","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:43.728485+0000","last_change":"2021-02-12T17:59:25.936443+0000","last_active":"2021-02-12T17:59:43.728485+0000","last_peered":"2021-02-12T17:59:43.728485+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:43.728485+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2119,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2856,"num_bytes_recovered":11694104,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4689}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":11220,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":22235,"num_bytes_recovered":91070488,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":460853248,"data_stored":460845104,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":10,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738402,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302884,"kb_used_data":215324,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068956,"statfs":{"total":96636764160,"available":96326610944,"internally_reserved":0,"allocated":220491776,"data_stored":220281036,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.32300000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.39200000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084295,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":221548,"kb_used_data":169060,"kb_used_omap":0,"kb_used_meta":52479,"kb_avail":94150292,"statfs":{"total":96636764160,"available":96409899008,"internally_reserved":0,"allocated":173117440,"data_stored":172906700,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":53738548},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]}]}} 2021-02-12T17:59:54.110 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:54.451 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:54.454 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:54.465 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2021-02-12T17:59:54.423529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23741,"num_bytes_recovered":97239064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":536372,"kb_used_data":389924,"kb_used_omap":0,"kb_used_meta":146431,"kb_avail":188207308,"statfs":{"total":193273528320,"available":192724283392,"internally_reserved":0,"allocated":399282176,"data_stored":398843626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":149945396},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2644,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001336"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753324+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:51.753324+0000","last_peered":"2021-02-12T17:59:51.753324+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:51.753324+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12502","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748988+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:51.748988+0000","last_peered":"2021-02-12T17:59:51.748988+0000","last_clean":"2021-02-12T17:59:51.748988+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:51.748988+0000","last_undegraded":"2021-02-12T17:59:51.748988+0000","last_fullsized":"2021-02-12T17:59:51.748988+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12589","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753030+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:51.753030+0000","last_peered":"2021-02-12T17:59:51.753030+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:51.753030+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12284","reported_epoch":"36","state":"active+clean","last_fresh":"2021-02-12T17:59:52.763903+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T17:59:52.763903+0000","last_peered":"2021-02-12T17:59:52.763903+0000","last_clean":"2021-02-12T17:59:52.763903+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T17:59:52.763903+0000","last_undegraded":"2021-02-12T17:59:52.763903+0000","last_fullsized":"2021-02-12T17:59:52.763903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12149","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753035+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:51.753035+0000","last_peered":"2021-02-12T17:59:51.753035+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:51.753035+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753160+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:51.753160+0000","last_peered":"2021-02-12T17:59:51.753160+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:51.753160+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12412","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748975+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:51.748975+0000","last_peered":"2021-02-12T17:59:51.748975+0000","last_clean":"2021-02-12T17:59:51.748975+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:51.748975+0000","last_undegraded":"2021-02-12T17:59:51.748975+0000","last_fullsized":"2021-02-12T17:59:51.748975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"13728","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:53.475176+0000","last_change":"2021-02-12T17:59:51.530536+0000","last_active":"2021-02-12T17:59:53.475176+0000","last_peered":"2021-02-12T17:59:53.475176+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:53.475176+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":14032920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23741,"num_bytes_recovered":97239064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466808832,"data_stored":466800688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302900,"kb_used_data":215340,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068940,"statfs":{"total":96636764160,"available":96326594560,"internally_reserved":0,"allocated":220508160,"data_stored":220286033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.33200000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233472,"kb_used_data":174584,"kb_used_omap":0,"kb_used_meta":58879,"kb_avail":94138368,"statfs":{"total":96636764160,"available":96397688832,"internally_reserved":0,"allocated":178774016,"data_stored":178557593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60292148},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]}]}]}} 2021-02-12T17:59:54.465 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:54.800 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:54.803 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:54.814 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":110,"stamp":"2021-02-12T17:59:54.423529+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23741,"num_bytes_recovered":97239064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":536372,"kb_used_data":389924,"kb_used_omap":0,"kb_used_meta":146431,"kb_avail":188207308,"statfs":{"total":193273528320,"available":192724283392,"internally_reserved":0,"allocated":399282176,"data_stored":398843626,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":149945396},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2644,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2644,"num_bytes_recovered":10829824,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001336"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753324+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:51.753324+0000","last_peered":"2021-02-12T17:59:51.753324+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:51.753324+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12502","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748988+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:51.748988+0000","last_peered":"2021-02-12T17:59:51.748988+0000","last_clean":"2021-02-12T17:59:51.748988+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:51.748988+0000","last_undegraded":"2021-02-12T17:59:51.748988+0000","last_fullsized":"2021-02-12T17:59:51.748988+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12589","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753030+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:51.753030+0000","last_peered":"2021-02-12T17:59:51.753030+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:51.753030+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12284","reported_epoch":"36","state":"active+clean","last_fresh":"2021-02-12T17:59:52.763903+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T17:59:52.763903+0000","last_peered":"2021-02-12T17:59:52.763903+0000","last_clean":"2021-02-12T17:59:52.763903+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T17:59:52.763903+0000","last_undegraded":"2021-02-12T17:59:52.763903+0000","last_fullsized":"2021-02-12T17:59:52.763903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12149","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753035+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:51.753035+0000","last_peered":"2021-02-12T17:59:51.753035+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:51.753035+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753160+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:51.753160+0000","last_peered":"2021-02-12T17:59:51.753160+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:51.753160+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12412","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748975+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:51.748975+0000","last_peered":"2021-02-12T17:59:51.748975+0000","last_clean":"2021-02-12T17:59:51.748975+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:51.748975+0000","last_undegraded":"2021-02-12T17:59:51.748975+0000","last_fullsized":"2021-02-12T17:59:51.748975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"13728","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:53.475176+0000","last_change":"2021-02-12T17:59:51.530536+0000","last_active":"2021-02-12T17:59:53.475176+0000","last_peered":"2021-02-12T17:59:53.475176+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:53.475176+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":14032920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23741,"num_bytes_recovered":97239064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":466808832,"data_stored":466800688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302900,"kb_used_data":215340,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068940,"statfs":{"total":96636764160,"available":96326594560,"internally_reserved":0,"allocated":220508160,"data_stored":220286033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.33200000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084296,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":233472,"kb_used_data":174584,"kb_used_omap":0,"kb_used_meta":58879,"kb_avail":94138368,"statfs":{"total":96636764160,"available":96397688832,"internally_reserved":0,"allocated":178774016,"data_stored":178557593,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60292148},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39000000000000001}]}]}]}} 2021-02-12T17:59:57.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:58.957 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:58.960 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:58.971 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2021-02-12T17:59:58.424134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23741,"num_bytes_recovered":97239064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":551968,"kb_used_data":395152,"kb_used_omap":0,"kb_used_meta":156799,"kb_avail":188191712,"statfs":{"total":193273528320,"available":192708313088,"internally_reserved":0,"allocated":404635648,"data_stored":404191394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160562228},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1506,"num_bytes_recovered":6168576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001322"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753324+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:51.753324+0000","last_peered":"2021-02-12T17:59:51.753324+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:51.753324+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12502","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748988+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:51.748988+0000","last_peered":"2021-02-12T17:59:51.748988+0000","last_clean":"2021-02-12T17:59:51.748988+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:51.748988+0000","last_undegraded":"2021-02-12T17:59:51.748988+0000","last_fullsized":"2021-02-12T17:59:51.748988+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12589","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753030+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:51.753030+0000","last_peered":"2021-02-12T17:59:51.753030+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:51.753030+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12284","reported_epoch":"36","state":"active+clean","last_fresh":"2021-02-12T17:59:52.763903+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T17:59:52.763903+0000","last_peered":"2021-02-12T17:59:52.763903+0000","last_clean":"2021-02-12T17:59:52.763903+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T17:59:52.763903+0000","last_undegraded":"2021-02-12T17:59:52.763903+0000","last_fullsized":"2021-02-12T17:59:52.763903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12149","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753035+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:51.753035+0000","last_peered":"2021-02-12T17:59:51.753035+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:51.753035+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753160+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:51.753160+0000","last_peered":"2021-02-12T17:59:51.753160+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:51.753160+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12412","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748975+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:51.748975+0000","last_peered":"2021-02-12T17:59:51.748975+0000","last_clean":"2021-02-12T17:59:51.748975+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:51.748975+0000","last_undegraded":"2021-02-12T17:59:51.748975+0000","last_fullsized":"2021-02-12T17:59:51.748975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"13728","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:53.475176+0000","last_change":"2021-02-12T17:59:51.530536+0000","last_active":"2021-02-12T17:59:53.475176+0000","last_peered":"2021-02-12T17:59:53.475176+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:53.475176+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":14032920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23741,"num_bytes_recovered":97239064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":472010752,"data_stored":472002608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302900,"kb_used_data":215340,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068940,"statfs":{"total":96636764160,"available":96326594560,"internally_reserved":0,"allocated":220508160,"data_stored":220286033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.33200000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249068,"kb_used_data":179812,"kb_used_omap":0,"kb_used_meta":69247,"kb_avail":94122772,"statfs":{"total":96636764160,"available":96381718528,"internally_reserved":0,"allocated":184127488,"data_stored":183905361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70908980},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]}]}]}} 2021-02-12T17:59:58.972 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T17:59:59.297 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T17:59:59.299 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T17:59:59.311 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":112,"stamp":"2021-02-12T17:59:58.424134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23741,"num_bytes_recovered":97239064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":551968,"kb_used_data":395152,"kb_used_omap":0,"kb_used_meta":156799,"kb_avail":188191712,"statfs":{"total":193273528320,"available":192708313088,"internally_reserved":0,"allocated":404635648,"data_stored":404191394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160562228},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1506,"num_bytes_recovered":6168576,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001322"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753324+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:51.753324+0000","last_peered":"2021-02-12T17:59:51.753324+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:51.753324+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12502","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748988+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:51.748988+0000","last_peered":"2021-02-12T17:59:51.748988+0000","last_clean":"2021-02-12T17:59:51.748988+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:51.748988+0000","last_undegraded":"2021-02-12T17:59:51.748988+0000","last_fullsized":"2021-02-12T17:59:51.748988+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12589","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753030+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:51.753030+0000","last_peered":"2021-02-12T17:59:51.753030+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:51.753030+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12284","reported_epoch":"36","state":"active+clean","last_fresh":"2021-02-12T17:59:52.763903+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T17:59:52.763903+0000","last_peered":"2021-02-12T17:59:52.763903+0000","last_clean":"2021-02-12T17:59:52.763903+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T17:59:52.763903+0000","last_undegraded":"2021-02-12T17:59:52.763903+0000","last_fullsized":"2021-02-12T17:59:52.763903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12149","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753035+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:51.753035+0000","last_peered":"2021-02-12T17:59:51.753035+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:51.753035+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753160+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:51.753160+0000","last_peered":"2021-02-12T17:59:51.753160+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:51.753160+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12412","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748975+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:51.748975+0000","last_peered":"2021-02-12T17:59:51.748975+0000","last_clean":"2021-02-12T17:59:51.748975+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:51.748975+0000","last_undegraded":"2021-02-12T17:59:51.748975+0000","last_fullsized":"2021-02-12T17:59:51.748975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"13728","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:53.475176+0000","last_change":"2021-02-12T17:59:51.530536+0000","last_active":"2021-02-12T17:59:53.475176+0000","last_peered":"2021-02-12T17:59:53.475176+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:53.475176+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1548,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3427,"num_bytes_recovered":14032920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5260}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":9714,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":23741,"num_bytes_recovered":97239064,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":472010752,"data_stored":472002608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738403,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302900,"kb_used_data":215340,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068940,"statfs":{"total":96636764160,"available":96326594560,"internally_reserved":0,"allocated":220508160,"data_stored":220286033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.33200000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249068,"kb_used_data":179812,"kb_used_omap":0,"kb_used_meta":69247,"kb_avail":94122772,"statfs":{"total":96636764160,"available":96381718528,"internally_reserved":0,"allocated":184127488,"data_stored":183905361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70908980},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]}]}]}} 2021-02-12T18:00:02.313 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:02.670 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:02.672 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:02.684 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2021-02-12T18:00:02.424705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24988,"num_bytes_recovered":102346776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":551968,"kb_used_data":395152,"kb_used_omap":0,"kb_used_meta":156799,"kb_avail":188191712,"statfs":{"total":193273528320,"available":192708313088,"internally_reserved":0,"allocated":404635648,"data_stored":404191394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160562228},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":5107712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001176"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753324+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:51.753324+0000","last_peered":"2021-02-12T17:59:51.753324+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:51.753324+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12502","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748988+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:51.748988+0000","last_peered":"2021-02-12T17:59:51.748988+0000","last_clean":"2021-02-12T17:59:51.748988+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:51.748988+0000","last_undegraded":"2021-02-12T17:59:51.748988+0000","last_fullsized":"2021-02-12T17:59:51.748988+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12589","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753030+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:51.753030+0000","last_peered":"2021-02-12T17:59:51.753030+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:51.753030+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12284","reported_epoch":"36","state":"active+clean","last_fresh":"2021-02-12T17:59:52.763903+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T17:59:52.763903+0000","last_peered":"2021-02-12T17:59:52.763903+0000","last_clean":"2021-02-12T17:59:52.763903+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T17:59:52.763903+0000","last_undegraded":"2021-02-12T17:59:52.763903+0000","last_fullsized":"2021-02-12T17:59:52.763903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12149","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753035+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:51.753035+0000","last_peered":"2021-02-12T17:59:51.753035+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:51.753035+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753160+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:51.753160+0000","last_peered":"2021-02-12T17:59:51.753160+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:51.753160+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12412","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748975+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:51.748975+0000","last_peered":"2021-02-12T17:59:51.748975+0000","last_clean":"2021-02-12T17:59:51.748975+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:51.748975+0000","last_undegraded":"2021-02-12T17:59:51.748975+0000","last_fullsized":"2021-02-12T17:59:51.748975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16222","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:58.474263+0000","last_change":"2021-02-12T17:59:51.530536+0000","last_active":"2021-02-12T17:59:58.474263+0000","last_peered":"2021-02-12T17:59:58.474263+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:58.474263+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4674,"num_bytes_recovered":19140632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6507}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24988,"num_bytes_recovered":102346776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":472010752,"data_stored":472002608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738404,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302900,"kb_used_data":215340,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068940,"statfs":{"total":96636764160,"available":96326594560,"internally_reserved":0,"allocated":220508160,"data_stored":220286033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.40300000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249068,"kb_used_data":179812,"kb_used_omap":0,"kb_used_meta":69247,"kb_avail":94122772,"statfs":{"total":96636764160,"available":96381718528,"internally_reserved":0,"allocated":184127488,"data_stored":183905361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70908980},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]}]}]}} 2021-02-12T18:00:02.685 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:03.021 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:03.023 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:03.035 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":116,"stamp":"2021-02-12T18:00:02.424705+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24988,"num_bytes_recovered":102346776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":551968,"kb_used_data":395152,"kb_used_omap":0,"kb_used_meta":156799,"kb_avail":188191712,"statfs":{"total":193273528320,"available":192708313088,"internally_reserved":0,"allocated":404635648,"data_stored":404191394,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160562228},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1247,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1247,"num_bytes_recovered":5107712,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001176"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753324+0000","last_change":"2021-02-12T17:59:25.941895+0000","last_active":"2021-02-12T17:59:51.753324+0000","last_peered":"2021-02-12T17:59:51.753324+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T17:59:51.753324+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1985,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4739}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12502","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748988+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:51.748988+0000","last_peered":"2021-02-12T17:59:51.748988+0000","last_clean":"2021-02-12T17:59:51.748988+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:51.748988+0000","last_undegraded":"2021-02-12T17:59:51.748988+0000","last_fullsized":"2021-02-12T17:59:51.748988+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12589","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753030+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:51.753030+0000","last_peered":"2021-02-12T17:59:51.753030+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:51.753030+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12284","reported_epoch":"36","state":"active+clean","last_fresh":"2021-02-12T17:59:52.763903+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T17:59:52.763903+0000","last_peered":"2021-02-12T17:59:52.763903+0000","last_clean":"2021-02-12T17:59:52.763903+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T17:59:52.763903+0000","last_undegraded":"2021-02-12T17:59:52.763903+0000","last_fullsized":"2021-02-12T17:59:52.763903+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12149","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753035+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:51.753035+0000","last_peered":"2021-02-12T17:59:51.753035+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:51.753035+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"35","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:51.753160+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:51.753160+0000","last_peered":"2021-02-12T17:59:51.753160+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:51.753160+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12412","reported_epoch":"35","state":"active+clean","last_fresh":"2021-02-12T17:59:51.748975+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:51.748975+0000","last_peered":"2021-02-12T17:59:51.748975+0000","last_clean":"2021-02-12T17:59:51.748975+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:51.748975+0000","last_undegraded":"2021-02-12T17:59:51.748975+0000","last_fullsized":"2021-02-12T17:59:51.748975+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16222","reported_epoch":"36","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:58.474263+0000","last_change":"2021-02-12T17:59:51.530536+0000","last_active":"2021-02-12T17:59:58.474263+0000","last_peered":"2021-02-12T17:59:58.474263+0000","last_clean":"2021-02-12T17:59:10.119396+0000","last_became_active":"2021-02-12T17:59:25.928096+0000","last_became_peered":"2021-02-12T17:59:25.928096+0000","last_unstale":"2021-02-12T17:59:58.474263+0000","last_undegraded":"2021-02-12T17:59:25.708095+0000","last_fullsized":"2021-02-12T17:59:25.662648+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":301,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4674,"num_bytes_recovered":19140632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6507}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":8467,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":24988,"num_bytes_recovered":102346776,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":472010752,"data_stored":472002608,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":11,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738404,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302900,"kb_used_data":215340,"kb_used_omap":0,"kb_used_meta":87552,"kb_avail":94068940,"statfs":{"total":96636764160,"available":96326594560,"internally_reserved":0,"allocated":220508160,"data_stored":220286033,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89653248},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.38500000000000001},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.40300000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084297,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":249068,"kb_used_data":179812,"kb_used_omap":0,"kb_used_meta":69247,"kb_avail":94122772,"statfs":{"total":96636764160,"available":96381718528,"internally_reserved":0,"allocated":184127488,"data_stored":183905361,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70908980},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.27300000000000002}]}]}]}} 2021-02-12T18:00:06.037 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:06.372 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:06.374 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:06.386 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":117,"stamp":"2021-02-12T18:00:04.424977+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26304,"num_bytes_recovered":107733040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":547680,"kb_used_data":400784,"kb_used_omap":0,"kb_used_meta":146879,"kb_avail":188196000,"statfs":{"total":193273528320,"available":192712704000,"internally_reserved":0,"allocated":410402816,"data_stored":409935688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":150404148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2563,"num_bytes_recovered":10493976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001112"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"8800","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:03.473493+0000","last_change":"2021-02-12T17:59:59.372977+0000","last_active":"2021-02-12T18:00:03.473493+0000","last_peered":"2021-02-12T18:00:03.473493+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T18:00:03.473493+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1015,"num_bytes_recovered":4157440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12504","reported_epoch":"37","state":"active+clean","last_fresh":"2021-02-12T17:59:59.765163+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:59.765163+0000","last_peered":"2021-02-12T17:59:59.765163+0000","last_clean":"2021-02-12T17:59:59.765163+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:59.765163+0000","last_undegraded":"2021-02-12T17:59:59.765163+0000","last_fullsized":"2021-02-12T17:59:59.765163+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12591","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.767712+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:59.767712+0000","last_peered":"2021-02-12T17:59:59.767712+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:59.767712+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12286","reported_epoch":"37","state":"active+clean","last_fresh":"2021-02-12T17:59:59.765084+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T17:59:59.765084+0000","last_peered":"2021-02-12T17:59:59.765084+0000","last_clean":"2021-02-12T17:59:59.765084+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T17:59:59.765084+0000","last_undegraded":"2021-02-12T17:59:59.765084+0000","last_fullsized":"2021-02-12T17:59:59.765084+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12151","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.767215+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:59.767215+0000","last_peered":"2021-02-12T17:59:59.767215+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:59.767215+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.768014+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:59.768014+0000","last_peered":"2021-02-12T17:59:59.768014+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:59.768014+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12414","reported_epoch":"37","state":"active+clean","last_fresh":"2021-02-12T17:59:59.765152+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:59.765152+0000","last_peered":"2021-02-12T17:59:59.765152+0000","last_clean":"2021-02-12T17:59:59.765152+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:59.765152+0000","last_undegraded":"2021-02-12T17:59:59.765152+0000","last_fullsized":"2021-02-12T17:59:59.765152+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16835","reported_epoch":"38","state":"active+clean","last_fresh":"2021-02-12T18:00:00.783894+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:00.783894+0000","last_peered":"2021-02-12T18:00:00.783894+0000","last_clean":"2021-02-12T18:00:00.783894+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:00.783894+0000","last_undegraded":"2021-02-12T18:00:00.783894+0000","last_fullsized":"2021-02-12T18:00:00.783894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26304,"num_bytes_recovered":107733040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":477540352,"data_stored":477532208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302980,"kb_used_data":215356,"kb_used_omap":0,"kb_used_meta":87616,"kb_avail":94068860,"statfs":{"total":96636764160,"available":96326512640,"internally_reserved":0,"allocated":220524544,"data_stored":220290980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89718784},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.38200000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244700,"kb_used_data":185428,"kb_used_omap":0,"kb_used_meta":59263,"kb_avail":94127140,"statfs":{"total":96636764160,"available":96386191360,"internally_reserved":0,"allocated":189878272,"data_stored":189644708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60685364},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]}]}} 2021-02-12T18:00:06.387 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:06.723 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:06.725 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:06.737 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":118,"stamp":"2021-02-12T18:00:06.425264+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26304,"num_bytes_recovered":107733040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":12,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":547680,"kb_used_data":400784,"kb_used_omap":0,"kb_used_meta":146879,"kb_avail":188196000,"statfs":{"total":193273528320,"available":192712704000,"internally_reserved":0,"allocated":410402816,"data_stored":409935688,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":150404148},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2563,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2563,"num_bytes_recovered":10493976,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001130"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"8800","reported_epoch":"38","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:03.473493+0000","last_change":"2021-02-12T17:59:59.372977+0000","last_active":"2021-02-12T18:00:03.473493+0000","last_peered":"2021-02-12T18:00:03.473493+0000","last_clean":"2021-02-12T17:59:10.107218+0000","last_became_active":"2021-02-12T17:59:25.934830+0000","last_became_peered":"2021-02-12T17:59:25.934830+0000","last_unstale":"2021-02-12T18:00:03.473493+0000","last_undegraded":"2021-02-12T17:59:25.702270+0000","last_fullsized":"2021-02-12T17:59:25.663288+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":970,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1015,"num_bytes_recovered":4157440,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5754}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12504","reported_epoch":"37","state":"active+clean","last_fresh":"2021-02-12T17:59:59.765163+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T17:59:59.765163+0000","last_peered":"2021-02-12T17:59:59.765163+0000","last_clean":"2021-02-12T17:59:59.765163+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T17:59:59.765163+0000","last_undegraded":"2021-02-12T17:59:59.765163+0000","last_fullsized":"2021-02-12T17:59:59.765163+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12591","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.767712+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:59.767712+0000","last_peered":"2021-02-12T17:59:59.767712+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:59.767712+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12286","reported_epoch":"37","state":"active+clean","last_fresh":"2021-02-12T17:59:59.765084+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T17:59:59.765084+0000","last_peered":"2021-02-12T17:59:59.765084+0000","last_clean":"2021-02-12T17:59:59.765084+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T17:59:59.765084+0000","last_undegraded":"2021-02-12T17:59:59.765084+0000","last_fullsized":"2021-02-12T17:59:59.765084+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12151","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.767215+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:59.767215+0000","last_peered":"2021-02-12T17:59:59.767215+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:59.767215+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.768014+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:59.768014+0000","last_peered":"2021-02-12T17:59:59.768014+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:59.768014+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12414","reported_epoch":"37","state":"active+clean","last_fresh":"2021-02-12T17:59:59.765152+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T17:59:59.765152+0000","last_peered":"2021-02-12T17:59:59.765152+0000","last_clean":"2021-02-12T17:59:59.765152+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T17:59:59.765152+0000","last_undegraded":"2021-02-12T17:59:59.765152+0000","last_fullsized":"2021-02-12T17:59:59.765152+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16835","reported_epoch":"38","state":"active+clean","last_fresh":"2021-02-12T18:00:00.783894+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:00.783894+0000","last_peered":"2021-02-12T18:00:00.783894+0000","last_clean":"2021-02-12T18:00:00.783894+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:00.783894+0000","last_undegraded":"2021-02-12T18:00:00.783894+0000","last_fullsized":"2021-02-12T18:00:00.783894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":7151,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":26304,"num_bytes_recovered":107733040,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":477540352,"data_stored":477532208,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":12,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302980,"kb_used_data":215356,"kb_used_omap":0,"kb_used_meta":87616,"kb_avail":94068860,"statfs":{"total":96636764160,"available":96326512640,"internally_reserved":0,"allocated":220524544,"data_stored":220290980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89718784},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.38200000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084298,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":244700,"kb_used_data":185428,"kb_used_omap":0,"kb_used_meta":59263,"kb_avail":94127140,"statfs":{"total":96636764160,"available":96386191360,"internally_reserved":0,"allocated":189878272,"data_stored":189644708,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":60685364},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.41999999999999998}]}]}]}} 2021-02-12T18:00:09.739 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:10.078 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:10.080 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:10.091 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":121,"stamp":"2021-02-12T18:00:08.425530+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27274,"num_bytes_recovered":111706160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":559512,"kb_used_data":406216,"kb_used_omap":0,"kb_used_meta":153279,"kb_avail":188184168,"statfs":{"total":193273528320,"available":192700588032,"internally_reserved":0,"allocated":415965184,"data_stored":415492323,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":156957748},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2286,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2286,"num_bytes_recovered":9359384,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"7.655334"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"40","state":"active+clean","last_fresh":"2021-02-12T18:00:07.801384+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:07.801384+0000","last_peered":"2021-02-12T18:00:07.801384+0000","last_clean":"2021-02-12T18:00:07.801384+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:07.801384+0000","last_undegraded":"2021-02-12T18:00:07.801384+0000","last_fullsized":"2021-02-12T18:00:07.801384+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12506","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785576+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:06.785576+0000","last_peered":"2021-02-12T18:00:06.785576+0000","last_clean":"2021-02-12T18:00:06.785576+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:06.785576+0000","last_undegraded":"2021-02-12T18:00:06.785576+0000","last_fullsized":"2021-02-12T18:00:06.785576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12591","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.767712+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T17:59:59.767712+0000","last_peered":"2021-02-12T17:59:59.767712+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T17:59:59.767712+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12288","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785560+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:06.785560+0000","last_peered":"2021-02-12T18:00:06.785560+0000","last_clean":"2021-02-12T18:00:06.785560+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:06.785560+0000","last_undegraded":"2021-02-12T18:00:06.785560+0000","last_fullsized":"2021-02-12T18:00:06.785560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12151","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.767215+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T17:59:59.767215+0000","last_peered":"2021-02-12T17:59:59.767215+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T17:59:59.767215+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"37","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T17:59:59.768014+0000","last_change":"2021-02-12T17:59:25.942055+0000","last_active":"2021-02-12T17:59:59.768014+0000","last_peered":"2021-02-12T17:59:59.768014+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T17:59:59.768014+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2039,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4648}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12416","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785581+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:06.785581+0000","last_peered":"2021-02-12T18:00:06.785581+0000","last_clean":"2021-02-12T18:00:06.785581+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:06.785581+0000","last_undegraded":"2021-02-12T18:00:06.785581+0000","last_fullsized":"2021-02-12T18:00:06.785581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16835","reported_epoch":"38","state":"active+clean","last_fresh":"2021-02-12T18:00:00.783894+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:00.783894+0000","last_peered":"2021-02-12T18:00:00.783894+0000","last_clean":"2021-02-12T18:00:00.783894+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:00.783894+0000","last_undegraded":"2021-02-12T18:00:00.783894+0000","last_fullsized":"2021-02-12T18:00:00.783894+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":6181,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27274,"num_bytes_recovered":111706160,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":482979840,"data_stored":482971696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":302980,"kb_used_data":215356,"kb_used_omap":0,"kb_used_meta":87616,"kb_avail":94068860,"statfs":{"total":96636764160,"available":96326512640,"internally_reserved":0,"allocated":220524544,"data_stored":220290980,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":89718784},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.29699999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.38200000000000001}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256532,"kb_used_data":190860,"kb_used_omap":0,"kb_used_meta":65663,"kb_avail":94115308,"statfs":{"total":96636764160,"available":96374075392,"internally_reserved":0,"allocated":195440640,"data_stored":195201343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":67238964},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]}]}} 2021-02-12T18:00:10.092 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:10.434 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:10.436 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:10.448 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":122,"stamp":"2021-02-12T18:00:10.425826+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27617,"num_bytes_recovered":113111088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":563624,"kb_used_data":406232,"kb_used_omap":0,"kb_used_meta":157375,"kb_avail":188180056,"statfs":{"total":193273528320,"available":192696377344,"internally_reserved":0,"allocated":415981568,"data_stored":415497220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":161152052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2629,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2629,"num_bytes_recovered":10764312,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001121"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"40","state":"active+clean","last_fresh":"2021-02-12T18:00:07.801384+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:07.801384+0000","last_peered":"2021-02-12T18:00:07.801384+0000","last_clean":"2021-02-12T18:00:07.801384+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:07.801384+0000","last_undegraded":"2021-02-12T18:00:07.801384+0000","last_fullsized":"2021-02-12T18:00:07.801384+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12506","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785576+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:06.785576+0000","last_peered":"2021-02-12T18:00:06.785576+0000","last_clean":"2021-02-12T18:00:06.785576+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:06.785576+0000","last_undegraded":"2021-02-12T18:00:06.785576+0000","last_fullsized":"2021-02-12T18:00:06.785576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12593","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788470+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T18:00:06.788470+0000","last_peered":"2021-02-12T18:00:06.788470+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:06.788470+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12288","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785560+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:06.785560+0000","last_peered":"2021-02-12T18:00:06.785560+0000","last_clean":"2021-02-12T18:00:06.785560+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:06.785560+0000","last_undegraded":"2021-02-12T18:00:06.785560+0000","last_fullsized":"2021-02-12T18:00:06.785560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12153","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788456+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:06.788456+0000","last_peered":"2021-02-12T18:00:06.788456+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:06.788456+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"7422","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:08.475499+0000","last_change":"2021-02-12T18:00:06.728043+0000","last_active":"2021-02-12T18:00:08.475499+0000","last_peered":"2021-02-12T18:00:08.475499+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T18:00:08.475499+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":1404928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4991}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12416","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785581+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:06.785581+0000","last_peered":"2021-02-12T18:00:06.785581+0000","last_clean":"2021-02-12T18:00:06.785581+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:06.785581+0000","last_undegraded":"2021-02-12T18:00:06.785581+0000","last_fullsized":"2021-02-12T18:00:06.785581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16837","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.788008+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:06.788008+0000","last_peered":"2021-02-12T18:00:06.788008+0000","last_clean":"2021-02-12T18:00:06.788008+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:06.788008+0000","last_undegraded":"2021-02-12T18:00:06.788008+0000","last_fullsized":"2021-02-12T18:00:06.788008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27617,"num_bytes_recovered":113111088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":482979840,"data_stored":482971696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307092,"kb_used_data":215372,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064748,"statfs":{"total":96636764160,"available":96322301952,"internally_reserved":0,"allocated":220540928,"data_stored":220295877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.442},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.47799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256532,"kb_used_data":190860,"kb_used_omap":0,"kb_used_meta":65663,"kb_avail":94115308,"statfs":{"total":96636764160,"available":96374075392,"internally_reserved":0,"allocated":195440640,"data_stored":195201343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":67238964},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]}]}} 2021-02-12T18:00:13.450 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:13.773 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:13.775 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:13.787 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2021-02-12T18:00:12.426127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27617,"num_bytes_recovered":113111088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":563624,"kb_used_data":406232,"kb_used_omap":0,"kb_used_meta":157375,"kb_avail":188180056,"statfs":{"total":193273528320,"available":192696377344,"internally_reserved":0,"allocated":415981568,"data_stored":415497220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":161152052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":5378048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001149"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"40","state":"active+clean","last_fresh":"2021-02-12T18:00:07.801384+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:07.801384+0000","last_peered":"2021-02-12T18:00:07.801384+0000","last_clean":"2021-02-12T18:00:07.801384+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:07.801384+0000","last_undegraded":"2021-02-12T18:00:07.801384+0000","last_fullsized":"2021-02-12T18:00:07.801384+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12506","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785576+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:06.785576+0000","last_peered":"2021-02-12T18:00:06.785576+0000","last_clean":"2021-02-12T18:00:06.785576+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:06.785576+0000","last_undegraded":"2021-02-12T18:00:06.785576+0000","last_fullsized":"2021-02-12T18:00:06.785576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12593","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788470+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T18:00:06.788470+0000","last_peered":"2021-02-12T18:00:06.788470+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:06.788470+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12288","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785560+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:06.785560+0000","last_peered":"2021-02-12T18:00:06.785560+0000","last_clean":"2021-02-12T18:00:06.785560+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:06.785560+0000","last_undegraded":"2021-02-12T18:00:06.785560+0000","last_fullsized":"2021-02-12T18:00:06.785560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12153","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788456+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:06.788456+0000","last_peered":"2021-02-12T18:00:06.788456+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:06.788456+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"7422","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:08.475499+0000","last_change":"2021-02-12T18:00:06.728043+0000","last_active":"2021-02-12T18:00:08.475499+0000","last_peered":"2021-02-12T18:00:08.475499+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T18:00:08.475499+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":1404928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4991}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12416","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785581+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:06.785581+0000","last_peered":"2021-02-12T18:00:06.785581+0000","last_clean":"2021-02-12T18:00:06.785581+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:06.785581+0000","last_undegraded":"2021-02-12T18:00:06.785581+0000","last_fullsized":"2021-02-12T18:00:06.785581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16837","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.788008+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:06.788008+0000","last_peered":"2021-02-12T18:00:06.788008+0000","last_clean":"2021-02-12T18:00:06.788008+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:06.788008+0000","last_undegraded":"2021-02-12T18:00:06.788008+0000","last_fullsized":"2021-02-12T18:00:06.788008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27617,"num_bytes_recovered":113111088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":482979840,"data_stored":482971696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307092,"kb_used_data":215372,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064748,"statfs":{"total":96636764160,"available":96322301952,"internally_reserved":0,"allocated":220540928,"data_stored":220295877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.442},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.47799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256532,"kb_used_data":190860,"kb_used_omap":0,"kb_used_meta":65663,"kb_avail":94115308,"statfs":{"total":96636764160,"available":96374075392,"internally_reserved":0,"allocated":195440640,"data_stored":195201343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":67238964},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]}]}} 2021-02-12T18:00:13.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:14.133 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:14.135 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:14.146 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":123,"stamp":"2021-02-12T18:00:12.426127+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27617,"num_bytes_recovered":113111088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":563624,"kb_used_data":406232,"kb_used_omap":0,"kb_used_meta":157375,"kb_avail":188180056,"statfs":{"total":193273528320,"available":192696377344,"internally_reserved":0,"allocated":415981568,"data_stored":415497220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":161152052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1313,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1313,"num_bytes_recovered":5378048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001149"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"40","state":"active+clean","last_fresh":"2021-02-12T18:00:07.801384+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:07.801384+0000","last_peered":"2021-02-12T18:00:07.801384+0000","last_clean":"2021-02-12T18:00:07.801384+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:07.801384+0000","last_undegraded":"2021-02-12T18:00:07.801384+0000","last_fullsized":"2021-02-12T18:00:07.801384+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12506","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785576+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:06.785576+0000","last_peered":"2021-02-12T18:00:06.785576+0000","last_clean":"2021-02-12T18:00:06.785576+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:06.785576+0000","last_undegraded":"2021-02-12T18:00:06.785576+0000","last_fullsized":"2021-02-12T18:00:06.785576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12593","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788470+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T18:00:06.788470+0000","last_peered":"2021-02-12T18:00:06.788470+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:06.788470+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12288","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785560+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:06.785560+0000","last_peered":"2021-02-12T18:00:06.785560+0000","last_clean":"2021-02-12T18:00:06.785560+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:06.785560+0000","last_undegraded":"2021-02-12T18:00:06.785560+0000","last_fullsized":"2021-02-12T18:00:06.785560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12153","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788456+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:06.788456+0000","last_peered":"2021-02-12T18:00:06.788456+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:06.788456+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"7422","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:08.475499+0000","last_change":"2021-02-12T18:00:06.728043+0000","last_active":"2021-02-12T18:00:08.475499+0000","last_peered":"2021-02-12T18:00:08.475499+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T18:00:08.475499+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1696,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":343,"num_bytes_recovered":1404928,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4991}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12416","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785581+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:06.785581+0000","last_peered":"2021-02-12T18:00:06.785581+0000","last_clean":"2021-02-12T18:00:06.785581+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:06.785581+0000","last_undegraded":"2021-02-12T18:00:06.785581+0000","last_fullsized":"2021-02-12T18:00:06.785581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16837","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.788008+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:06.788008+0000","last_peered":"2021-02-12T18:00:06.788008+0000","last_clean":"2021-02-12T18:00:06.788008+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:06.788008+0000","last_undegraded":"2021-02-12T18:00:06.788008+0000","last_fullsized":"2021-02-12T18:00:06.788008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":5838,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":27617,"num_bytes_recovered":113111088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":482979840,"data_stored":482971696,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738406,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307092,"kb_used_data":215372,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064748,"statfs":{"total":96636764160,"available":96322301952,"internally_reserved":0,"allocated":220540928,"data_stored":220295877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.442},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.47799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084299,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":256532,"kb_used_data":190860,"kb_used_omap":0,"kb_used_meta":65663,"kb_avail":94115308,"statfs":{"total":96636764160,"available":96374075392,"internally_reserved":0,"allocated":195440640,"data_stored":195201343,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":67238964},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]}]}} 2021-02-12T18:00:17.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:17.476 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:17.478 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:17.489 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2021-02-12T18:00:16.830813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28712,"num_bytes_recovered":117596208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":573704,"kb_used_data":410488,"kb_used_omap":0,"kb_used_meta":163199,"kb_avail":188169976,"statfs":{"total":193273528320,"available":192686055424,"internally_reserved":0,"allocated":420339712,"data_stored":419849610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167115828},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":5890048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.405282"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"40","state":"active+clean","last_fresh":"2021-02-12T18:00:07.801384+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:07.801384+0000","last_peered":"2021-02-12T18:00:07.801384+0000","last_clean":"2021-02-12T18:00:07.801384+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:07.801384+0000","last_undegraded":"2021-02-12T18:00:07.801384+0000","last_fullsized":"2021-02-12T18:00:07.801384+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12506","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785576+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:06.785576+0000","last_peered":"2021-02-12T18:00:06.785576+0000","last_clean":"2021-02-12T18:00:06.785576+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:06.785576+0000","last_undegraded":"2021-02-12T18:00:06.785576+0000","last_fullsized":"2021-02-12T18:00:06.785576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12593","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788470+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T18:00:06.788470+0000","last_peered":"2021-02-12T18:00:06.788470+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:06.788470+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12288","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785560+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:06.785560+0000","last_peered":"2021-02-12T18:00:06.785560+0000","last_clean":"2021-02-12T18:00:06.785560+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:06.785560+0000","last_undegraded":"2021-02-12T18:00:06.785560+0000","last_fullsized":"2021-02-12T18:00:06.785560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12153","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788456+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:06.788456+0000","last_peered":"2021-02-12T18:00:06.788456+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:06.788456+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"9612","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:13.321421+0000","last_change":"2021-02-12T18:00:06.728043+0000","last_active":"2021-02-12T18:00:13.321421+0000","last_peered":"2021-02-12T18:00:13.321421+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T18:00:13.321421+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":5890048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12416","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785581+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:06.785581+0000","last_peered":"2021-02-12T18:00:06.785581+0000","last_clean":"2021-02-12T18:00:06.785581+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:06.785581+0000","last_undegraded":"2021-02-12T18:00:06.785581+0000","last_fullsized":"2021-02-12T18:00:06.785581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16837","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.788008+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:06.788008+0000","last_peered":"2021-02-12T18:00:06.788008+0000","last_clean":"2021-02-12T18:00:06.788008+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:06.788008+0000","last_undegraded":"2021-02-12T18:00:06.788008+0000","last_fullsized":"2021-02-12T18:00:06.788008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28712,"num_bytes_recovered":117596208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487505920,"data_stored":487497776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307092,"kb_used_data":215372,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064748,"statfs":{"total":96636764160,"available":96322301952,"internally_reserved":0,"allocated":220540928,"data_stored":220295877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.36399999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266612,"kb_used_data":195116,"kb_used_omap":0,"kb_used_meta":71487,"kb_avail":94105228,"statfs":{"total":96636764160,"available":96363753472,"internally_reserved":0,"allocated":199798784,"data_stored":199553733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73202740},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]}]}} 2021-02-12T18:00:17.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:17.828 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:17.830 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:17.842 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":127,"stamp":"2021-02-12T18:00:16.830813+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28712,"num_bytes_recovered":117596208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":573704,"kb_used_data":410488,"kb_used_omap":0,"kb_used_meta":163199,"kb_avail":188169976,"statfs":{"total":193273528320,"available":192686055424,"internally_reserved":0,"allocated":420339712,"data_stored":419849610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167115828},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1438,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":5890048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.405282"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6767","reported_epoch":"40","state":"active+clean","last_fresh":"2021-02-12T18:00:07.801384+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:07.801384+0000","last_peered":"2021-02-12T18:00:07.801384+0000","last_clean":"2021-02-12T18:00:07.801384+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:07.801384+0000","last_undegraded":"2021-02-12T18:00:07.801384+0000","last_fullsized":"2021-02-12T18:00:07.801384+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12506","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785576+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:06.785576+0000","last_peered":"2021-02-12T18:00:06.785576+0000","last_clean":"2021-02-12T18:00:06.785576+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:06.785576+0000","last_undegraded":"2021-02-12T18:00:06.785576+0000","last_fullsized":"2021-02-12T18:00:06.785576+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"12593","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788470+0000","last_change":"2021-02-12T17:59:25.948324+0000","last_active":"2021-02-12T18:00:06.788470+0000","last_peered":"2021-02-12T18:00:06.788470+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:06.788470+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2166,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2857,"num_bytes_recovered":11702272,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4650}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12288","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785560+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:06.785560+0000","last_peered":"2021-02-12T18:00:06.785560+0000","last_clean":"2021-02-12T18:00:06.785560+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:06.785560+0000","last_undegraded":"2021-02-12T18:00:06.785560+0000","last_fullsized":"2021-02-12T18:00:06.785560+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12153","reported_epoch":"39","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:06.788456+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:06.788456+0000","last_peered":"2021-02-12T18:00:06.788456+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:06.788456+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"9612","reported_epoch":"40","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:13.321421+0000","last_change":"2021-02-12T18:00:06.728043+0000","last_active":"2021-02-12T18:00:13.321421+0000","last_peered":"2021-02-12T18:00:13.321421+0000","last_clean":"2021-02-12T17:59:10.107840+0000","last_became_active":"2021-02-12T17:59:25.934671+0000","last_became_peered":"2021-02-12T17:59:25.934671+0000","last_unstale":"2021-02-12T18:00:13.321421+0000","last_undegraded":"2021-02-12T17:59:25.706078+0000","last_fullsized":"2021-02-12T17:59:25.662482+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":601,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1438,"num_bytes_recovered":5890048,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6086}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12416","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.785581+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:06.785581+0000","last_peered":"2021-02-12T18:00:06.785581+0000","last_clean":"2021-02-12T18:00:06.785581+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:06.785581+0000","last_undegraded":"2021-02-12T18:00:06.785581+0000","last_fullsized":"2021-02-12T18:00:06.785581+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16837","reported_epoch":"39","state":"active+clean","last_fresh":"2021-02-12T18:00:06.788008+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:06.788008+0000","last_peered":"2021-02-12T18:00:06.788008+0000","last_clean":"2021-02-12T18:00:06.788008+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:06.788008+0000","last_undegraded":"2021-02-12T18:00:06.788008+0000","last_fullsized":"2021-02-12T18:00:06.788008+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":4743,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":28712,"num_bytes_recovered":117596208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":487505920,"data_stored":487497776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":13,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738407,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307092,"kb_used_data":215372,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064748,"statfs":{"total":96636764160,"available":96322301952,"internally_reserved":0,"allocated":220540928,"data_stored":220295877,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[1],"upper_bound":2},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.30299999999999999},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.36399999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084300,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":266612,"kb_used_data":195116,"kb_used_omap":0,"kb_used_meta":71487,"kb_avail":94105228,"statfs":{"total":96636764160,"available":96363753472,"internally_reserved":0,"allocated":199798784,"data_stored":199553733,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73202740},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36799999999999999}]}]}]}} 2021-02-12T18:00:20.844 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:21.183 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:21.185 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:21.197 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2021-02-12T18:00:20.427267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29831,"num_bytes_recovered":122179632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":572296,"kb_used_data":415288,"kb_used_omap":0,"kb_used_meta":156991,"kb_avail":188171384,"statfs":{"total":193273528320,"available":192687497216,"internally_reserved":0,"allocated":425254912,"data_stored":424741736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160758836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2214,"num_bytes_recovered":9068544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001139"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6769","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826468+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:15.826468+0000","last_peered":"2021-02-12T18:00:15.826468+0000","last_clean":"2021-02-12T18:00:15.826468+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:15.826468+0000","last_undegraded":"2021-02-12T18:00:15.826468+0000","last_fullsized":"2021-02-12T18:00:15.826468+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12508","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826180+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:15.826180+0000","last_peered":"2021-02-12T18:00:15.826180+0000","last_clean":"2021-02-12T18:00:15.826180+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:15.826180+0000","last_undegraded":"2021-02-12T18:00:15.826180+0000","last_fullsized":"2021-02-12T18:00:15.826180+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"13632","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:18.477259+0000","last_change":"2021-02-12T18:00:15.526810+0000","last_active":"2021-02-12T18:00:18.477259+0000","last_peered":"2021-02-12T18:00:18.477259+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:18.477259+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3375,"num_bytes_recovered":13824000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12290","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826032+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:15.826032+0000","last_peered":"2021-02-12T18:00:15.826032+0000","last_clean":"2021-02-12T18:00:15.826032+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:15.826032+0000","last_undegraded":"2021-02-12T18:00:15.826032+0000","last_fullsized":"2021-02-12T18:00:15.826032+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12155","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:15.828298+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:15.828298+0000","last_peered":"2021-02-12T18:00:15.828298+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:15.828298+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"42","state":"active+clean","last_fresh":"2021-02-12T18:00:16.843341+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:16.843341+0000","last_peered":"2021-02-12T18:00:16.843341+0000","last_clean":"2021-02-12T18:00:16.843341+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:16.843341+0000","last_undegraded":"2021-02-12T18:00:16.843341+0000","last_fullsized":"2021-02-12T18:00:16.843341+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12418","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826083+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:15.826083+0000","last_peered":"2021-02-12T18:00:15.826083+0000","last_clean":"2021-02-12T18:00:15.826083+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:15.826083+0000","last_undegraded":"2021-02-12T18:00:15.826083+0000","last_fullsized":"2021-02-12T18:00:15.826083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16839","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.827394+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:15.827394+0000","last_peered":"2021-02-12T18:00:15.827394+0000","last_clean":"2021-02-12T18:00:15.827394+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:15.827394+0000","last_undegraded":"2021-02-12T18:00:15.827394+0000","last_fullsized":"2021-02-12T18:00:15.827394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29831,"num_bytes_recovered":122179632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":492122112,"data_stored":492113968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307108,"kb_used_data":215388,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064732,"statfs":{"total":96636764160,"available":96322285568,"internally_reserved":0,"allocated":220557312,"data_stored":220300724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.371},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.35299999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265188,"kb_used_data":199900,"kb_used_omap":0,"kb_used_meta":65279,"kb_avail":94106652,"statfs":{"total":96636764160,"available":96365211648,"internally_reserved":0,"allocated":204697600,"data_stored":204441012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":66845748},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]}]}} 2021-02-12T18:00:21.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:21.527 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:21.529 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:21.540 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":129,"stamp":"2021-02-12T18:00:20.427267+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29831,"num_bytes_recovered":122179632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":572296,"kb_used_data":415288,"kb_used_omap":0,"kb_used_meta":156991,"kb_avail":188171384,"statfs":{"total":193273528320,"available":192687497216,"internally_reserved":0,"allocated":425254912,"data_stored":424741736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160758836},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2214,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2214,"num_bytes_recovered":9068544,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001139"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6769","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826468+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:15.826468+0000","last_peered":"2021-02-12T18:00:15.826468+0000","last_clean":"2021-02-12T18:00:15.826468+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:15.826468+0000","last_undegraded":"2021-02-12T18:00:15.826468+0000","last_fullsized":"2021-02-12T18:00:15.826468+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12508","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826180+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:15.826180+0000","last_peered":"2021-02-12T18:00:15.826180+0000","last_clean":"2021-02-12T18:00:15.826180+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:15.826180+0000","last_undegraded":"2021-02-12T18:00:15.826180+0000","last_fullsized":"2021-02-12T18:00:15.826180+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"13632","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:18.477259+0000","last_change":"2021-02-12T18:00:15.526810+0000","last_active":"2021-02-12T18:00:18.477259+0000","last_peered":"2021-02-12T18:00:18.477259+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:18.477259+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1648,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3375,"num_bytes_recovered":13824000,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5168}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12290","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826032+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:15.826032+0000","last_peered":"2021-02-12T18:00:15.826032+0000","last_clean":"2021-02-12T18:00:15.826032+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:15.826032+0000","last_undegraded":"2021-02-12T18:00:15.826032+0000","last_fullsized":"2021-02-12T18:00:15.826032+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12155","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:15.828298+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:15.828298+0000","last_peered":"2021-02-12T18:00:15.828298+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:15.828298+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"42","state":"active+clean","last_fresh":"2021-02-12T18:00:16.843341+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:16.843341+0000","last_peered":"2021-02-12T18:00:16.843341+0000","last_clean":"2021-02-12T18:00:16.843341+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:16.843341+0000","last_undegraded":"2021-02-12T18:00:16.843341+0000","last_fullsized":"2021-02-12T18:00:16.843341+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12418","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826083+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:15.826083+0000","last_peered":"2021-02-12T18:00:15.826083+0000","last_clean":"2021-02-12T18:00:15.826083+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:15.826083+0000","last_undegraded":"2021-02-12T18:00:15.826083+0000","last_fullsized":"2021-02-12T18:00:15.826083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16839","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.827394+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:15.827394+0000","last_peered":"2021-02-12T18:00:15.827394+0000","last_clean":"2021-02-12T18:00:15.827394+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:15.827394+0000","last_undegraded":"2021-02-12T18:00:15.827394+0000","last_fullsized":"2021-02-12T18:00:15.827394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":3624,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":29831,"num_bytes_recovered":122179632,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":492122112,"data_stored":492113968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738408,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307108,"kb_used_data":215388,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064732,"statfs":{"total":96636764160,"available":96322285568,"internally_reserved":0,"allocated":220557312,"data_stored":220300724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.371},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.35299999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084301,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":265188,"kb_used_data":199900,"kb_used_omap":0,"kb_used_meta":65279,"kb_avail":94106652,"statfs":{"total":96636764160,"available":96365211648,"internally_reserved":0,"allocated":204697600,"data_stored":204441012,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":66845748},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.32900000000000001}]}]}]}} 2021-02-12T18:00:24.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:24.883 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:24.885 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:24.896 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2021-02-12T18:00:24.427842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30949,"num_bytes_recovered":126758960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":582028,"kb_used_data":419256,"kb_used_omap":0,"kb_used_meta":162751,"kb_avail":188161652,"statfs":{"total":193273528320,"available":192677531648,"internally_reserved":0,"allocated":429318144,"data_stored":428804968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":166657076},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":9162752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.605118"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6769","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826468+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:15.826468+0000","last_peered":"2021-02-12T18:00:15.826468+0000","last_clean":"2021-02-12T18:00:15.826468+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:15.826468+0000","last_undegraded":"2021-02-12T18:00:15.826468+0000","last_fullsized":"2021-02-12T18:00:15.826468+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12508","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826180+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:15.826180+0000","last_peered":"2021-02-12T18:00:15.826180+0000","last_clean":"2021-02-12T18:00:15.826180+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:15.826180+0000","last_undegraded":"2021-02-12T18:00:15.826180+0000","last_fullsized":"2021-02-12T18:00:15.826180+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"15868","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:23.478760+0000","last_change":"2021-02-12T18:00:15.526810+0000","last_active":"2021-02-12T18:00:23.478760+0000","last_peered":"2021-02-12T18:00:23.478760+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:23.478760+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4493,"num_bytes_recovered":18403328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6286}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12290","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826032+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:15.826032+0000","last_peered":"2021-02-12T18:00:15.826032+0000","last_clean":"2021-02-12T18:00:15.826032+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:15.826032+0000","last_undegraded":"2021-02-12T18:00:15.826032+0000","last_fullsized":"2021-02-12T18:00:15.826032+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12155","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:15.828298+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:15.828298+0000","last_peered":"2021-02-12T18:00:15.828298+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:15.828298+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"42","state":"active+clean","last_fresh":"2021-02-12T18:00:16.843341+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:16.843341+0000","last_peered":"2021-02-12T18:00:16.843341+0000","last_clean":"2021-02-12T18:00:16.843341+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:16.843341+0000","last_undegraded":"2021-02-12T18:00:16.843341+0000","last_fullsized":"2021-02-12T18:00:16.843341+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12418","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826083+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:15.826083+0000","last_peered":"2021-02-12T18:00:15.826083+0000","last_clean":"2021-02-12T18:00:15.826083+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:15.826083+0000","last_undegraded":"2021-02-12T18:00:15.826083+0000","last_fullsized":"2021-02-12T18:00:15.826083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16839","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.827394+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:15.827394+0000","last_peered":"2021-02-12T18:00:15.827394+0000","last_clean":"2021-02-12T18:00:15.827394+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:15.827394+0000","last_undegraded":"2021-02-12T18:00:15.827394+0000","last_fullsized":"2021-02-12T18:00:15.827394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30949,"num_bytes_recovered":126758960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":496492544,"data_stored":496484400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307108,"kb_used_data":215388,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064732,"statfs":{"total":96636764160,"available":96322285568,"internally_reserved":0,"allocated":220557312,"data_stored":220300724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.39400000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":274920,"kb_used_data":203868,"kb_used_omap":0,"kb_used_meta":71039,"kb_avail":94096920,"statfs":{"total":96636764160,"available":96355246080,"internally_reserved":0,"allocated":208760832,"data_stored":208504244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":72743988},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]}]}} 2021-02-12T18:00:24.897 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:25.246 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:25.248 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:25.259 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":131,"stamp":"2021-02-12T18:00:24.427842+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30949,"num_bytes_recovered":126758960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":582028,"kb_used_data":419256,"kb_used_omap":0,"kb_used_meta":162751,"kb_avail":188161652,"statfs":{"total":193273528320,"available":192677531648,"internally_reserved":0,"allocated":429318144,"data_stored":428804968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":166657076},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2237,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2237,"num_bytes_recovered":9162752,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.605118"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6769","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826468+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:15.826468+0000","last_peered":"2021-02-12T18:00:15.826468+0000","last_clean":"2021-02-12T18:00:15.826468+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:15.826468+0000","last_undegraded":"2021-02-12T18:00:15.826468+0000","last_fullsized":"2021-02-12T18:00:15.826468+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12508","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826180+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:15.826180+0000","last_peered":"2021-02-12T18:00:15.826180+0000","last_clean":"2021-02-12T18:00:15.826180+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:15.826180+0000","last_undegraded":"2021-02-12T18:00:15.826180+0000","last_fullsized":"2021-02-12T18:00:15.826180+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"15868","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:23.478760+0000","last_change":"2021-02-12T18:00:15.526810+0000","last_active":"2021-02-12T18:00:23.478760+0000","last_peered":"2021-02-12T18:00:23.478760+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:23.478760+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4493,"num_bytes_recovered":18403328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6286}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12290","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826032+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:15.826032+0000","last_peered":"2021-02-12T18:00:15.826032+0000","last_clean":"2021-02-12T18:00:15.826032+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:15.826032+0000","last_undegraded":"2021-02-12T18:00:15.826032+0000","last_fullsized":"2021-02-12T18:00:15.826032+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12155","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:15.828298+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:15.828298+0000","last_peered":"2021-02-12T18:00:15.828298+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:15.828298+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6731","reported_epoch":"42","state":"active+clean","last_fresh":"2021-02-12T18:00:16.843341+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:16.843341+0000","last_peered":"2021-02-12T18:00:16.843341+0000","last_clean":"2021-02-12T18:00:16.843341+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:16.843341+0000","last_undegraded":"2021-02-12T18:00:16.843341+0000","last_fullsized":"2021-02-12T18:00:16.843341+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12418","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.826083+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:15.826083+0000","last_peered":"2021-02-12T18:00:15.826083+0000","last_clean":"2021-02-12T18:00:15.826083+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:15.826083+0000","last_undegraded":"2021-02-12T18:00:15.826083+0000","last_fullsized":"2021-02-12T18:00:15.826083+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16839","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.827394+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:15.827394+0000","last_peered":"2021-02-12T18:00:15.827394+0000","last_clean":"2021-02-12T18:00:15.827394+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:15.827394+0000","last_undegraded":"2021-02-12T18:00:15.827394+0000","last_fullsized":"2021-02-12T18:00:15.827394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30949,"num_bytes_recovered":126758960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":496492544,"data_stored":496484400,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307108,"kb_used_data":215388,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064732,"statfs":{"total":96636764160,"available":96322285568,"internally_reserved":0,"allocated":220557312,"data_stored":220300724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.39400000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084302,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":274920,"kb_used_data":203868,"kb_used_omap":0,"kb_used_meta":71039,"kb_avail":94096920,"statfs":{"total":96636764160,"available":96355246080,"internally_reserved":0,"allocated":208760832,"data_stored":208504244,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":72743988},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,1],"upper_bound":4},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45500000000000002}]}]}]}} 2021-02-12T18:00:28.261 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:28.590 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:28.592 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:28.603 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2021-02-12T18:00:28.428422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30949,"num_bytes_recovered":126758960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":593564,"kb_used_data":424524,"kb_used_omap":0,"kb_used_meta":169023,"kb_avail":188150116,"statfs":{"total":193273528320,"available":192665718784,"internally_reserved":0,"allocated":434712576,"data_stored":434187813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173079604},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":4579328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001154"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6771","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866433+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:25.866433+0000","last_peered":"2021-02-12T18:00:25.866433+0000","last_clean":"2021-02-12T18:00:25.866433+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:25.866433+0000","last_undegraded":"2021-02-12T18:00:25.866433+0000","last_fullsized":"2021-02-12T18:00:25.866433+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12510","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866342+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:25.866342+0000","last_peered":"2021-02-12T18:00:25.866342+0000","last_clean":"2021-02-12T18:00:25.866342+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:25.866342+0000","last_undegraded":"2021-02-12T18:00:25.866342+0000","last_fullsized":"2021-02-12T18:00:25.866342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"15868","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:23.478760+0000","last_change":"2021-02-12T18:00:15.526810+0000","last_active":"2021-02-12T18:00:23.478760+0000","last_peered":"2021-02-12T18:00:23.478760+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:23.478760+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4493,"num_bytes_recovered":18403328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6286}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12292","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866200+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:25.866200+0000","last_peered":"2021-02-12T18:00:25.866200+0000","last_clean":"2021-02-12T18:00:25.866200+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:25.866200+0000","last_undegraded":"2021-02-12T18:00:25.866200+0000","last_fullsized":"2021-02-12T18:00:25.866200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12155","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:15.828298+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:15.828298+0000","last_peered":"2021-02-12T18:00:15.828298+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:15.828298+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866204+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:25.866204+0000","last_peered":"2021-02-12T18:00:25.866204+0000","last_clean":"2021-02-12T18:00:25.866204+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:25.866204+0000","last_undegraded":"2021-02-12T18:00:25.866204+0000","last_fullsized":"2021-02-12T18:00:25.866204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12420","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866244+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:25.866244+0000","last_peered":"2021-02-12T18:00:25.866244+0000","last_clean":"2021-02-12T18:00:25.866244+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:25.866244+0000","last_undegraded":"2021-02-12T18:00:25.866244+0000","last_fullsized":"2021-02-12T18:00:25.866244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16839","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.827394+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:15.827394+0000","last_peered":"2021-02-12T18:00:15.827394+0000","last_clean":"2021-02-12T18:00:15.827394+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:15.827394+0000","last_undegraded":"2021-02-12T18:00:15.827394+0000","last_fullsized":"2021-02-12T18:00:15.827394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30949,"num_bytes_recovered":126758960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501719040,"data_stored":501710896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307108,"kb_used_data":215388,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064732,"statfs":{"total":96636764160,"available":96322285568,"internally_reserved":0,"allocated":220557312,"data_stored":220300724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.39400000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286456,"kb_used_data":209136,"kb_used_omap":0,"kb_used_meta":77311,"kb_avail":94085384,"statfs":{"total":96636764160,"available":96343433216,"internally_reserved":0,"allocated":214155264,"data_stored":213887089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":79166516},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38100000000000001}]}]}]}} 2021-02-12T18:00:28.604 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:28.961 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:28.963 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:28.974 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":135,"stamp":"2021-02-12T18:00:28.428422+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30949,"num_bytes_recovered":126758960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":593564,"kb_used_data":424524,"kb_used_omap":0,"kb_used_meta":169023,"kb_avail":188150116,"statfs":{"total":193273528320,"available":192665718784,"internally_reserved":0,"allocated":434712576,"data_stored":434187813,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173079604},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1118,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1118,"num_bytes_recovered":4579328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001154"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6771","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866433+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:25.866433+0000","last_peered":"2021-02-12T18:00:25.866433+0000","last_clean":"2021-02-12T18:00:25.866433+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:25.866433+0000","last_undegraded":"2021-02-12T18:00:25.866433+0000","last_fullsized":"2021-02-12T18:00:25.866433+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12510","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866342+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:25.866342+0000","last_peered":"2021-02-12T18:00:25.866342+0000","last_clean":"2021-02-12T18:00:25.866342+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:25.866342+0000","last_undegraded":"2021-02-12T18:00:25.866342+0000","last_fullsized":"2021-02-12T18:00:25.866342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"15868","reported_epoch":"42","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:23.478760+0000","last_change":"2021-02-12T18:00:15.526810+0000","last_active":"2021-02-12T18:00:23.478760+0000","last_peered":"2021-02-12T18:00:23.478760+0000","last_clean":"2021-02-12T17:59:10.130872+0000","last_became_active":"2021-02-12T17:59:25.939342+0000","last_became_peered":"2021-02-12T17:59:25.939342+0000","last_unstale":"2021-02-12T18:00:23.478760+0000","last_undegraded":"2021-02-12T17:59:25.703673+0000","last_fullsized":"2021-02-12T17:59:25.662640+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":18,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":530,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4493,"num_bytes_recovered":18403328,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6286}],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12292","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866200+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:25.866200+0000","last_peered":"2021-02-12T18:00:25.866200+0000","last_clean":"2021-02-12T18:00:25.866200+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:25.866200+0000","last_undegraded":"2021-02-12T18:00:25.866200+0000","last_fullsized":"2021-02-12T18:00:25.866200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12155","reported_epoch":"41","state":"active+recovery_wait+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:15.828298+0000","last_change":"2021-02-12T17:59:25.948936+0000","last_active":"2021-02-12T18:00:15.828298+0000","last_peered":"2021-02-12T18:00:15.828298+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:15.828298+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1976,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2779,"num_bytes_recovered":11382784,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":4557}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866204+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:25.866204+0000","last_peered":"2021-02-12T18:00:25.866204+0000","last_clean":"2021-02-12T18:00:25.866204+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:25.866204+0000","last_undegraded":"2021-02-12T18:00:25.866204+0000","last_fullsized":"2021-02-12T18:00:25.866204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12420","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866244+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:25.866244+0000","last_peered":"2021-02-12T18:00:25.866244+0000","last_clean":"2021-02-12T18:00:25.866244+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:25.866244+0000","last_undegraded":"2021-02-12T18:00:25.866244+0000","last_fullsized":"2021-02-12T18:00:25.866244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16839","reported_epoch":"41","state":"active+clean","last_fresh":"2021-02-12T18:00:15.827394+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:15.827394+0000","last_peered":"2021-02-12T18:00:15.827394+0000","last_clean":"2021-02-12T18:00:15.827394+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:15.827394+0000","last_undegraded":"2021-02-12T18:00:15.827394+0000","last_fullsized":"2021-02-12T18:00:15.827394+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":2506,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":30949,"num_bytes_recovered":126758960,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501719040,"data_stored":501710896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":14,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307108,"kb_used_data":215388,"kb_used_omap":0,"kb_used_meta":91712,"kb_avail":94064732,"statfs":{"total":96636764160,"available":96322285568,"internally_reserved":0,"allocated":220557312,"data_stored":220300724,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93913088},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 17:58:24 2021","interfaces":[{"interface":"back","average":{"1min":1.0069999999999999,"5min":1.0069999999999999,"15min":1.0069999999999999},"min":{"1min":0.28699999999999998,"5min":0.28699999999999998,"15min":0.28699999999999998},"max":{"1min":6.827,"5min":6.827,"15min":6.827},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.626,"5min":0.626,"15min":0.626},"min":{"1min":0.29699999999999999,"5min":0.29699999999999999,"15min":0.29699999999999999},"max":{"1min":2.4580000000000002,"5min":2.4580000000000002,"15min":2.4580000000000002},"last":0.39400000000000002}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286456,"kb_used_data":209136,"kb_used_omap":0,"kb_used_meta":77311,"kb_avail":94085384,"statfs":{"total":96636764160,"available":96343433216,"internally_reserved":0,"allocated":214155264,"data_stored":213887089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":79166516},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38100000000000001}]}]}]}} 2021-02-12T18:00:31.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:32.551 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:32.552 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:32.563 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2021-02-12T18:00:32.429084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32185,"num_bytes_recovered":131821616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":593644,"kb_used_data":424540,"kb_used_omap":0,"kb_used_meta":169087,"kb_avail":188150036,"statfs":{"total":193273528320,"available":192665636864,"internally_reserved":0,"allocated":434728960,"data_stored":434192610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173145140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":5062656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001241"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6771","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866433+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:25.866433+0000","last_peered":"2021-02-12T18:00:25.866433+0000","last_clean":"2021-02-12T18:00:25.866433+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:25.866433+0000","last_undegraded":"2021-02-12T18:00:25.866433+0000","last_fullsized":"2021-02-12T18:00:25.866433+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12510","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866342+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:25.866342+0000","last_peered":"2021-02-12T18:00:25.866342+0000","last_clean":"2021-02-12T18:00:25.866342+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:25.866342+0000","last_undegraded":"2021-02-12T18:00:25.866342+0000","last_fullsized":"2021-02-12T18:00:25.866342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16939","reported_epoch":"44","state":"active+clean","last_fresh":"2021-02-12T18:00:26.881505+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:26.881505+0000","last_peered":"2021-02-12T18:00:26.881505+0000","last_clean":"2021-02-12T18:00:26.881505+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:26.881505+0000","last_undegraded":"2021-02-12T18:00:26.881505+0000","last_fullsized":"2021-02-12T18:00:26.881505+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12292","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866200+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:25.866200+0000","last_peered":"2021-02-12T18:00:25.866200+0000","last_clean":"2021-02-12T18:00:25.866200+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:25.866200+0000","last_undegraded":"2021-02-12T18:00:25.866200+0000","last_fullsized":"2021-02-12T18:00:25.866200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"13570","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:28.476457+0000","last_change":"2021-02-12T18:00:25.194779+0000","last_active":"2021-02-12T18:00:28.476457+0000","last_peered":"2021-02-12T18:00:28.476457+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:28.476457+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3485,"num_bytes_recovered":14274560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5263}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866204+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:25.866204+0000","last_peered":"2021-02-12T18:00:25.866204+0000","last_clean":"2021-02-12T18:00:25.866204+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:25.866204+0000","last_undegraded":"2021-02-12T18:00:25.866204+0000","last_fullsized":"2021-02-12T18:00:25.866204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12420","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866244+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:25.866244+0000","last_peered":"2021-02-12T18:00:25.866244+0000","last_clean":"2021-02-12T18:00:25.866244+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:25.866244+0000","last_undegraded":"2021-02-12T18:00:25.866244+0000","last_fullsized":"2021-02-12T18:00:25.866244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16841","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.867065+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:25.867065+0000","last_peered":"2021-02-12T18:00:25.867065+0000","last_clean":"2021-02-12T18:00:25.867065+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:25.867065+0000","last_undegraded":"2021-02-12T18:00:25.867065+0000","last_fullsized":"2021-02-12T18:00:25.867065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32185,"num_bytes_recovered":131821616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501719040,"data_stored":501710896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307188,"kb_used_data":215404,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064652,"statfs":{"total":96636764160,"available":96322203648,"internally_reserved":0,"allocated":220573696,"data_stored":220305521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.376},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.47799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286456,"kb_used_data":209136,"kb_used_omap":0,"kb_used_meta":77311,"kb_avail":94085384,"statfs":{"total":96636764160,"available":96343433216,"internally_reserved":0,"allocated":214155264,"data_stored":213887089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":79166516},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38100000000000001}]}]}]}} 2021-02-12T18:00:32.564 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:32.893 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:32.895 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:32.906 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":137,"stamp":"2021-02-12T18:00:32.429084+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32185,"num_bytes_recovered":131821616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":593644,"kb_used_data":424540,"kb_used_omap":0,"kb_used_meta":169087,"kb_avail":188150036,"statfs":{"total":193273528320,"available":192665636864,"internally_reserved":0,"allocated":434728960,"data_stored":434192610,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173145140},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1236,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1236,"num_bytes_recovered":5062656,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001241"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6771","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866433+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:25.866433+0000","last_peered":"2021-02-12T18:00:25.866433+0000","last_clean":"2021-02-12T18:00:25.866433+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:25.866433+0000","last_undegraded":"2021-02-12T18:00:25.866433+0000","last_fullsized":"2021-02-12T18:00:25.866433+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12510","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866342+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:25.866342+0000","last_peered":"2021-02-12T18:00:25.866342+0000","last_clean":"2021-02-12T18:00:25.866342+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:25.866342+0000","last_undegraded":"2021-02-12T18:00:25.866342+0000","last_fullsized":"2021-02-12T18:00:25.866342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16939","reported_epoch":"44","state":"active+clean","last_fresh":"2021-02-12T18:00:26.881505+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:26.881505+0000","last_peered":"2021-02-12T18:00:26.881505+0000","last_clean":"2021-02-12T18:00:26.881505+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:26.881505+0000","last_undegraded":"2021-02-12T18:00:26.881505+0000","last_fullsized":"2021-02-12T18:00:26.881505+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12292","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866200+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:25.866200+0000","last_peered":"2021-02-12T18:00:25.866200+0000","last_clean":"2021-02-12T18:00:25.866200+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:25.866200+0000","last_undegraded":"2021-02-12T18:00:25.866200+0000","last_fullsized":"2021-02-12T18:00:25.866200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"13570","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:28.476457+0000","last_change":"2021-02-12T18:00:25.194779+0000","last_active":"2021-02-12T18:00:28.476457+0000","last_peered":"2021-02-12T18:00:28.476457+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:28.476457+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":3485,"num_bytes_recovered":14274560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":5263}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866204+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:25.866204+0000","last_peered":"2021-02-12T18:00:25.866204+0000","last_clean":"2021-02-12T18:00:25.866204+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:25.866204+0000","last_undegraded":"2021-02-12T18:00:25.866204+0000","last_fullsized":"2021-02-12T18:00:25.866204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12420","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866244+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:25.866244+0000","last_peered":"2021-02-12T18:00:25.866244+0000","last_clean":"2021-02-12T18:00:25.866244+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:25.866244+0000","last_undegraded":"2021-02-12T18:00:25.866244+0000","last_fullsized":"2021-02-12T18:00:25.866244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16841","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.867065+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:25.867065+0000","last_peered":"2021-02-12T18:00:25.867065+0000","last_clean":"2021-02-12T18:00:25.867065+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:25.867065+0000","last_undegraded":"2021-02-12T18:00:25.867065+0000","last_fullsized":"2021-02-12T18:00:25.867065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":32185,"num_bytes_recovered":131821616,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":501719040,"data_stored":501710896,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738410,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307188,"kb_used_data":215404,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064652,"statfs":{"total":96636764160,"available":96322203648,"internally_reserved":0,"allocated":220573696,"data_stored":220305521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.376},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.47799999999999998}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084303,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":286456,"kb_used_data":209136,"kb_used_omap":0,"kb_used_meta":77311,"kb_avail":94085384,"statfs":{"total":96636764160,"available":96343433216,"internally_reserved":0,"allocated":214155264,"data_stored":213887089,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":79166516},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.27700000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.38100000000000001}]}]}]}} 2021-02-12T18:00:35.908 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:36.211 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:36.213 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:36.224 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":140,"stamp":"2021-02-12T18:00:34.896904+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33374,"num_bytes_recovered":136691760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":587568,"kb_used_data":428960,"kb_used_omap":0,"kb_used_meta":158591,"kb_avail":188156112,"statfs":{"total":193273528320,"available":192671858688,"internally_reserved":0,"allocated":439255040,"data_stored":438718690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":162397236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2425,"num_bytes_recovered":9932800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.026191"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6771","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866433+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:25.866433+0000","last_peered":"2021-02-12T18:00:25.866433+0000","last_clean":"2021-02-12T18:00:25.866433+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:25.866433+0000","last_undegraded":"2021-02-12T18:00:25.866433+0000","last_fullsized":"2021-02-12T18:00:25.866433+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12510","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866342+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:25.866342+0000","last_peered":"2021-02-12T18:00:25.866342+0000","last_clean":"2021-02-12T18:00:25.866342+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:25.866342+0000","last_undegraded":"2021-02-12T18:00:25.866342+0000","last_fullsized":"2021-02-12T18:00:25.866342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16939","reported_epoch":"44","state":"active+clean","last_fresh":"2021-02-12T18:00:26.881505+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:26.881505+0000","last_peered":"2021-02-12T18:00:26.881505+0000","last_clean":"2021-02-12T18:00:26.881505+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:26.881505+0000","last_undegraded":"2021-02-12T18:00:26.881505+0000","last_fullsized":"2021-02-12T18:00:26.881505+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12292","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866200+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:25.866200+0000","last_peered":"2021-02-12T18:00:25.866200+0000","last_clean":"2021-02-12T18:00:25.866200+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:25.866200+0000","last_undegraded":"2021-02-12T18:00:25.866200+0000","last_fullsized":"2021-02-12T18:00:25.866200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"15948","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:33.478465+0000","last_change":"2021-02-12T18:00:25.194779+0000","last_active":"2021-02-12T18:00:33.478465+0000","last_peered":"2021-02-12T18:00:33.478465+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:33.478465+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4674,"num_bytes_recovered":19144704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6452}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866204+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:25.866204+0000","last_peered":"2021-02-12T18:00:25.866204+0000","last_clean":"2021-02-12T18:00:25.866204+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:25.866204+0000","last_undegraded":"2021-02-12T18:00:25.866204+0000","last_fullsized":"2021-02-12T18:00:25.866204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12420","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866244+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:25.866244+0000","last_peered":"2021-02-12T18:00:25.866244+0000","last_clean":"2021-02-12T18:00:25.866244+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:25.866244+0000","last_undegraded":"2021-02-12T18:00:25.866244+0000","last_fullsized":"2021-02-12T18:00:25.866244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16841","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.867065+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:25.867065+0000","last_peered":"2021-02-12T18:00:25.867065+0000","last_clean":"2021-02-12T18:00:25.867065+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:25.867065+0000","last_undegraded":"2021-02-12T18:00:25.867065+0000","last_fullsized":"2021-02-12T18:00:25.867065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33374,"num_bytes_recovered":136691760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":506286080,"data_stored":506277936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307188,"kb_used_data":215404,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064652,"statfs":{"total":96636764160,"available":96322203648,"internally_reserved":0,"allocated":220573696,"data_stored":220305521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.442},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.36299999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280380,"kb_used_data":213556,"kb_used_omap":0,"kb_used_meta":66815,"kb_avail":94091460,"statfs":{"total":96636764160,"available":96349655040,"internally_reserved":0,"allocated":218681344,"data_stored":218413169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":68418612},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.47499999999999998}]}]}]}} 2021-02-12T18:00:36.225 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:36.514 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:36.515 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:36.526 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":141,"stamp":"2021-02-12T18:00:36.429827+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33374,"num_bytes_recovered":136691760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":15,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":587568,"kb_used_data":428960,"kb_used_omap":0,"kb_used_meta":158591,"kb_avail":188156112,"statfs":{"total":193273528320,"available":192671858688,"internally_reserved":0,"allocated":439255040,"data_stored":438718690,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":162397236},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-2425,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2425,"num_bytes_recovered":9932800,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001405"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6771","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866433+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:25.866433+0000","last_peered":"2021-02-12T18:00:25.866433+0000","last_clean":"2021-02-12T18:00:25.866433+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:25.866433+0000","last_undegraded":"2021-02-12T18:00:25.866433+0000","last_fullsized":"2021-02-12T18:00:25.866433+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12510","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866342+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:25.866342+0000","last_peered":"2021-02-12T18:00:25.866342+0000","last_clean":"2021-02-12T18:00:25.866342+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:25.866342+0000","last_undegraded":"2021-02-12T18:00:25.866342+0000","last_fullsized":"2021-02-12T18:00:25.866342+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16939","reported_epoch":"44","state":"active+clean","last_fresh":"2021-02-12T18:00:26.881505+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:26.881505+0000","last_peered":"2021-02-12T18:00:26.881505+0000","last_clean":"2021-02-12T18:00:26.881505+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:26.881505+0000","last_undegraded":"2021-02-12T18:00:26.881505+0000","last_fullsized":"2021-02-12T18:00:26.881505+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12292","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866200+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:25.866200+0000","last_peered":"2021-02-12T18:00:25.866200+0000","last_clean":"2021-02-12T18:00:25.866200+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:25.866200+0000","last_undegraded":"2021-02-12T18:00:25.866200+0000","last_fullsized":"2021-02-12T18:00:25.866200+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"15948","reported_epoch":"44","state":"active+recovering+undersized+degraded+remapped","last_fresh":"2021-02-12T18:00:33.478465+0000","last_change":"2021-02-12T18:00:25.194779+0000","last_active":"2021-02-12T18:00:33.478465+0000","last_peered":"2021-02-12T18:00:33.478465+0000","last_clean":"2021-02-12T17:59:10.112882+0000","last_became_active":"2021-02-12T17:59:25.941864+0000","last_became_peered":"2021-02-12T17:59:25.941864+0000","last_unstale":"2021-02-12T18:00:33.478465+0000","last_undegraded":"2021-02-12T17:59:25.708713+0000","last_fullsized":"2021-02-12T17:59:25.662442+0000","mapping_epoch":29,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":16,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4674,"num_bytes_recovered":19144704,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[2],"avail_no_missing":["2"],"object_location_counts":[{"shards":"2","objects":6452}],"blocked_by":[],"up_primary":1,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6733","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866204+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:25.866204+0000","last_peered":"2021-02-12T18:00:25.866204+0000","last_clean":"2021-02-12T18:00:25.866204+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:25.866204+0000","last_undegraded":"2021-02-12T18:00:25.866204+0000","last_fullsized":"2021-02-12T18:00:25.866204+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12420","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.866244+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:25.866244+0000","last_peered":"2021-02-12T18:00:25.866244+0000","last_clean":"2021-02-12T18:00:25.866244+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:25.866244+0000","last_undegraded":"2021-02-12T18:00:25.866244+0000","last_fullsized":"2021-02-12T18:00:25.866244+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16841","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.867065+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:25.867065+0000","last_peered":"2021-02-12T18:00:25.867065+0000","last_clean":"2021-02-12T18:00:25.867065+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:25.867065+0000","last_undegraded":"2021-02-12T18:00:25.867065+0000","last_fullsized":"2021-02-12T18:00:25.867065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":81,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33374,"num_bytes_recovered":136691760,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":506286080,"data_stored":506277936,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":15,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307188,"kb_used_data":215404,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064652,"statfs":{"total":96636764160,"available":96322203648,"internally_reserved":0,"allocated":220573696,"data_stored":220305521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.442},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.36299999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084304,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":280380,"kb_used_data":213556,"kb_used_omap":0,"kb_used_meta":66815,"kb_avail":94091460,"statfs":{"total":96636764160,"available":96349655040,"internally_reserved":0,"allocated":218681344,"data_stored":218413169,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":68418612},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.47499999999999998}]}]}]}} 2021-02-12T18:00:39.530 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:39.833 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:39.835 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:39.846 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":142,"stamp":"2021-02-12T18:00:38.430152+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594360,"kb_used_data":430824,"kb_used_omap":0,"kb_used_meta":163519,"kb_avail":188149320,"statfs":{"total":193273528320,"available":192664903680,"internally_reserved":0,"allocated":441163776,"data_stored":440615789,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167443508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":-1270,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1270,"num_bytes_recovered":5201920,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.001437"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6773","reported_epoch":"45","state":"active+clean","last_fresh":"2021-02-12T18:00:33.888241+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:33.888241+0000","last_peered":"2021-02-12T18:00:33.888241+0000","last_clean":"2021-02-12T18:00:33.888241+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:33.888241+0000","last_undegraded":"2021-02-12T18:00:33.888241+0000","last_fullsized":"2021-02-12T18:00:33.888241+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12512","reported_epoch":"45","state":"active+clean","last_fresh":"2021-02-12T18:00:33.888289+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:33.888289+0000","last_peered":"2021-02-12T18:00:33.888289+0000","last_clean":"2021-02-12T18:00:33.888289+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:33.888289+0000","last_undegraded":"2021-02-12T18:00:33.888289+0000","last_fullsized":"2021-02-12T18:00:33.888289+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16939","reported_epoch":"44","state":"active+clean","last_fresh":"2021-02-12T18:00:26.881505+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:26.881505+0000","last_peered":"2021-02-12T18:00:26.881505+0000","last_clean":"2021-02-12T18:00:26.881505+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:26.881505+0000","last_undegraded":"2021-02-12T18:00:26.881505+0000","last_fullsized":"2021-02-12T18:00:26.881505+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12294","reported_epoch":"45","state":"active+clean","last_fresh":"2021-02-12T18:00:33.888185+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:33.888185+0000","last_peered":"2021-02-12T18:00:33.888185+0000","last_clean":"2021-02-12T18:00:33.888185+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:33.888185+0000","last_undegraded":"2021-02-12T18:00:33.888185+0000","last_fullsized":"2021-02-12T18:00:33.888185+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12149","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.908986+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.908986+0000","last_peered":"2021-02-12T18:00:34.908986+0000","last_clean":"2021-02-12T18:00:34.908986+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.908986+0000","last_undegraded":"2021-02-12T18:00:34.908986+0000","last_fullsized":"2021-02-12T18:00:34.908986+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6735","reported_epoch":"45","state":"active+clean","last_fresh":"2021-02-12T18:00:33.888234+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:33.888234+0000","last_peered":"2021-02-12T18:00:33.888234+0000","last_clean":"2021-02-12T18:00:33.888234+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:33.888234+0000","last_undegraded":"2021-02-12T18:00:33.888234+0000","last_fullsized":"2021-02-12T18:00:33.888234+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12422","reported_epoch":"45","state":"active+clean","last_fresh":"2021-02-12T18:00:33.888195+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:33.888195+0000","last_peered":"2021-02-12T18:00:33.888195+0000","last_clean":"2021-02-12T18:00:33.888195+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:33.888195+0000","last_undegraded":"2021-02-12T18:00:33.888195+0000","last_fullsized":"2021-02-12T18:00:33.888195+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16841","reported_epoch":"43","state":"active+clean","last_fresh":"2021-02-12T18:00:25.867065+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:25.867065+0000","last_peered":"2021-02-12T18:00:25.867065+0000","last_clean":"2021-02-12T18:00:25.867065+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:25.867065+0000","last_undegraded":"2021-02-12T18:00:25.867065+0000","last_fullsized":"2021-02-12T18:00:25.867065+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":507408384,"data_stored":507400240,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738411,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307188,"kb_used_data":215404,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064652,"statfs":{"total":96636764160,"available":96322203648,"internally_reserved":0,"allocated":220573696,"data_stored":220305521,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61799999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.442},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.36299999999999999}]}]},{"osd":0,"up_from":0,"seq":0,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084305,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287172,"kb_used_data":215420,"kb_used_omap":0,"kb_used_meta":71743,"kb_avail":94084668,"statfs":{"total":96636764160,"available":96342700032,"internally_reserved":0,"allocated":220590080,"data_stored":220310268,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73464884},"hb_peers":[2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.45200000000000001},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.67800000000000005}]}]}]}} 2021-02-12T18:00:39.847 INFO:tasks.osd_backfill.ceph_manager:recovered! 2021-02-12T18:00:39.847 INFO:tasks.ceph.osd.0:Restarting daemon 2021-02-12T18:00:39.848 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2021-02-12T18:00:39.850 INFO:tasks.ceph.osd.0:Started 2021-02-12T18:00:39.850 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2021-02-12T18:00:39.972 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2021-02-12T18:00:39.972 INFO:teuthology.orchestra.run.smithi107.stderr:0 2021-02-12T18:00:39.972 INFO:teuthology.orchestra.run.smithi107.stderr:1 2021-02-12T18:00:39.972 INFO:teuthology.orchestra.run.smithi107.stderr:2 2021-02-12T18:00:39.972 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2021-02-12T18:00:39.973 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2021-02-12T18:00:39.973 INFO:teuthology.orchestra.run.smithi107.stderr:config diff 2021-02-12T18:00:39.973 INFO:teuthology.orchestra.run.smithi107.stderr:config diff get 2021-02-12T18:00:39.973 INFO:teuthology.orchestra.run.smithi107.stderr:config get 2021-02-12T18:00:39.973 INFO:teuthology.orchestra.run.smithi107.stderr:config help [] 2021-02-12T18:00:39.973 INFO:teuthology.orchestra.run.smithi107.stderr:config set ... 2021-02-12T18:00:39.974 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2021-02-12T18:00:39.974 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-12T18:00:39.975 INFO:tasks.osd_backfill.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2021-02-12T18:00:40.437 INFO:tasks.ceph.osd.0.smithi107.stderr:2021-02-12T18:00:40.436+0000 7f35de833080 -1 Falling back to public interface 2021-02-12T18:00:44.976 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:no valid command found; 10 closest matches: 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:0 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:1 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:2 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:abort 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:assert 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs files list 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:bluefs stats 2021-02-12T18:00:45.101 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator dump block 2021-02-12T18:00:45.102 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator fragmentation block 2021-02-12T18:00:45.102 INFO:teuthology.orchestra.run.smithi107.stderr:bluestore allocator score block 2021-02-12T18:00:45.102 INFO:teuthology.orchestra.run.smithi107.stderr:admin_socket: invalid command 2021-02-12T18:00:45.103 DEBUG:teuthology.orchestra.run:got remote process result: 22 2021-02-12T18:00:45.103 INFO:tasks.osd_backfill.ceph_manager:waiting on admin_socket for osd-0, ['dump_ops_in_flight'] 2021-02-12T18:00:45.358 INFO:tasks.ceph.osd.0.smithi107.stderr:2021-02-12T18:00:45.356+0000 7f35de833080 -1 osd.0 12 log_to_monitors {default=true} 2021-02-12T18:00:50.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 0 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-osd.0.asok dump_ops_in_flight 2021-02-12T18:00:50.240 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2021-02-12T18:00:50.241 INFO:teuthology.orchestra.run.smithi107.stdout: "ops": [], 2021-02-12T18:00:50.241 INFO:teuthology.orchestra.run.smithi107.stdout: "num_ops": 0 2021-02-12T18:00:50.241 INFO:teuthology.orchestra.run.smithi107.stdout:} 2021-02-12T18:00:50.250 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2021-02-12T18:00:50.400 INFO:teuthology.orchestra.run.smithi107.stdout:120259084308 2021-02-12T18:00:50.401 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2021-02-12T18:00:50.550 INFO:teuthology.orchestra.run.smithi107.stdout:34359738415 2021-02-12T18:00:50.551 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-12T18:00:50.843 INFO:teuthology.orchestra.run.smithi107.stdout:120259084307 2021-02-12T18:00:50.856 INFO:tasks.osd_backfill.ceph_manager:need seq 120259084308 got 120259084307 for osd.1 2021-02-12T18:00:51.857 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2021-02-12T18:00:52.159 INFO:teuthology.orchestra.run.smithi107.stdout:120259084308 2021-02-12T18:00:52.172 INFO:tasks.osd_backfill.ceph_manager:need seq 120259084308 got 120259084308 for osd.1 2021-02-12T18:00:52.172 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-12T18:00:52.466 INFO:teuthology.orchestra.run.smithi107.stdout:34359738414 2021-02-12T18:00:52.478 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738415 got 34359738414 for osd.2 2021-02-12T18:00:53.479 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2021-02-12T18:00:53.777 INFO:teuthology.orchestra.run.smithi107.stdout:34359738415 2021-02-12T18:00:53.789 INFO:tasks.osd_backfill.ceph_manager:need seq 34359738415 got 34359738415 for osd.2 2021-02-12T18:00:53.789 INFO:tasks.osd_backfill.ceph_manager:waiting for clean 2021-02-12T18:00:53.790 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:54.080 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:54.082 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:54.093 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2021-02-12T18:00:52.432700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594392,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163519,"kb_avail":188149288,"statfs":{"total":193273528320,"available":192664870912,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167443508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001839"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12295","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899186+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:34.899186+0000","last_peered":"2021-02-12T18:00:34.899186+0000","last_clean":"2021-02-12T18:00:34.899186+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:34.899186+0000","last_undegraded":"2021-02-12T18:00:34.899186+0000","last_fullsized":"2021-02-12T18:00:34.899186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":505982976,"data_stored":505970760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738415,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307212,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064628,"statfs":{"total":96636764160,"available":96322179072,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.82899999999999996}]}]},{"osd":0,"up_from":0,"seq":201863462914,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084308,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287180,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":71743,"kb_avail":94084660,"statfs":{"total":96636764160,"available":96342691840,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73464884},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.69999999999999996}]}]}]}} 2021-02-12T18:00:54.094 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:54.389 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:54.391 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:54.401 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":150,"stamp":"2021-02-12T18:00:52.432700+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594392,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163519,"kb_avail":188149288,"statfs":{"total":193273528320,"available":192664870912,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167443508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001839"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12295","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899186+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:34.899186+0000","last_peered":"2021-02-12T18:00:34.899186+0000","last_clean":"2021-02-12T18:00:34.899186+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:34.899186+0000","last_undegraded":"2021-02-12T18:00:34.899186+0000","last_fullsized":"2021-02-12T18:00:34.899186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":505982976,"data_stored":505970760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738415,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307212,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064628,"statfs":{"total":96636764160,"available":96322179072,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67200000000000004}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.82899999999999996}]}]},{"osd":0,"up_from":0,"seq":201863462914,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084308,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287180,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":71743,"kb_avail":94084660,"statfs":{"total":96636764160,"available":96342691840,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73464884},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.69999999999999996}]}]}]}} 2021-02-12T18:00:54.402 INFO:tasks.osd_backfill.ceph_manager:clean! 2021-02-12T18:00:54.402 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2021-02-12T18:00:54.411 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2021-02-12T18:00:54.411 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-02-12T18:00:54.703 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:54.703 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":47,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T18:00:45.962618+0000","last_up_change":"2021-02-12T18:00:45.962618+0000","last_in_change":"2021-02-12T17:57:50.561433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-12T17:57:23.210299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":12,"up_from":47,"up_thru":9,"down_at":13,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":42490}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":42490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":42490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":42490}]},"public_addr":"172.21.15.107:6804/42490","cluster_addr":"172.21.15.107:6805/42490","heartbeat_back_addr":"172.21.15.107:6807/42490","heartbeat_front_addr":"172.21.15.107:6806/42490","state":["exists","up"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":28,"up_thru":45,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":39656}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":39656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":39656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":39656}]},"public_addr":"172.21.15.107:6800/39656","cluster_addr":"172.21.15.107:6801/39656","heartbeat_back_addr":"172.21.15.107:6803/39656","heartbeat_front_addr":"172.21.15.107:6802/39656","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":65536,"last_purged_snaps_scrub":"2021-02-12T17:57:20.700614+0000","dead_epoch":13},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.801932+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.752083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T18:00:54.716 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:55.006 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:55.008 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:55.018 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":151,"stamp":"2021-02-12T18:00:54.433025+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594392,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163519,"kb_avail":188149288,"statfs":{"total":193273528320,"available":192664870912,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167443508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.001777"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12295","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899186+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:34.899186+0000","last_peered":"2021-02-12T18:00:34.899186+0000","last_clean":"2021-02-12T18:00:34.899186+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:34.899186+0000","last_undegraded":"2021-02-12T18:00:34.899186+0000","last_fullsized":"2021-02-12T18:00:34.899186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":505982976,"data_stored":505970760,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738416,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307212,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064628,"statfs":{"total":96636764160,"available":96322179072,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.69699999999999995}]}]},{"osd":0,"up_from":0,"seq":201863462914,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084309,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287180,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":71743,"kb_avail":94084660,"statfs":{"total":96636764160,"available":96342691840,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73464884},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47599999999999998}]},{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54600000000000004},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.69999999999999996}]}]}]}} 2021-02-12T18:00:55.019 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2021-02-12T18:00:55.314 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:55.314 INFO:teuthology.orchestra.run.smithi107.stdout:{"epoch":47,"fsid":"9fbe23e3-d93f-4ea0-a0ec-b3e12a35eb0c","created":"2021-02-12T17:57:13.894357+0000","modified":"2021-02-12T18:00:45.962618+0000","last_up_change":"2021-02-12T18:00:45.962618+0000","last_in_change":"2021-02-12T17:57:50.561433+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":"rbd","create_time":"2021-02-12T17:57:23.210299+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"c10bf242-c6be-4d2a-817c-901bb5e8f07a","up":1,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":12,"up_from":47,"up_thru":9,"down_at":13,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6804","nonce":42490}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6805","nonce":42490}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6807","nonce":42490}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6806","nonce":42490}]},"public_addr":"172.21.15.107:6804/42490","cluster_addr":"172.21.15.107:6805/42490","heartbeat_back_addr":"172.21.15.107:6807/42490","heartbeat_front_addr":"172.21.15.107:6806/42490","state":["exists","up"]},{"osd":1,"uuid":"b0ff20b4-ad6a-46a3-a5dd-e2929ba97d9d","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":8,"last_clean_end":25,"up_from":28,"up_thru":45,"down_at":26,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6800","nonce":39656}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6801","nonce":39656}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6803","nonce":39656}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6802","nonce":39656}]},"public_addr":"172.21.15.107:6800/39656","cluster_addr":"172.21.15.107:6801/39656","heartbeat_back_addr":"172.21.15.107:6803/39656","heartbeat_front_addr":"172.21.15.107:6802/39656","state":["exists","up"]},{"osd":2,"uuid":"ab2b9420-4808-48df-b9c1-8683e619c87a","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":44,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6808","nonce":33988}]},"cluster_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6809","nonce":33988}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6811","nonce":33988}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v1","addr":"172.21.15.107:6810","nonce":33988}]},"public_addr":"172.21.15.107:6808/33988","cluster_addr":"172.21.15.107:6809/33988","heartbeat_back_addr":"172.21.15.107:6811/33988","heartbeat_front_addr":"172.21.15.107:6810/33988","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":65536,"last_purged_snaps_scrub":"2021-02-12T17:57:20.700614+0000","dead_epoch":13},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.801932+0000","dead_epoch":26},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138303579357183,"old_weight":0,"last_purged_snaps_scrub":"2021-02-12T17:57:20.752083+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2021-02-12T18:00:56.328 INFO:tasks.ceph:Scrubbing osd.0 2021-02-12T18:00:56.328 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2021-02-12T18:00:56.479 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2021-02-12T18:00:56.479 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-02-12T18:00:56.479 INFO:teuthology.orchestra.run.smithi107.stdout:} 2021-02-12T18:00:56.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2021-02-12T18:00:56.787 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 0 to deep-scrub 2021-02-12T18:00:57.197 INFO:tasks.ceph:Scrubbing osd.1 2021-02-12T18:00:57.198 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2021-02-12T18:00:57.340 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2021-02-12T18:00:57.340 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-02-12T18:00:57.340 INFO:teuthology.orchestra.run.smithi107.stdout:} 2021-02-12T18:00:57.351 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2021-02-12T18:00:57.652 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 1 to deep-scrub 2021-02-12T18:00:57.662 INFO:tasks.ceph:Scrubbing osd.2 2021-02-12T18:00:57.662 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2021-02-12T18:00:57.803 INFO:teuthology.orchestra.run.smithi107.stdout:{ 2021-02-12T18:00:57.803 INFO:teuthology.orchestra.run.smithi107.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2021-02-12T18:00:57.804 INFO:teuthology.orchestra.run.smithi107.stdout:} 2021-02-12T18:00:57.814 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2021-02-12T18:00:58.117 INFO:teuthology.orchestra.run.smithi107.stderr:instructed osd(s) 2 to deep-scrub 2021-02-12T18:00:58.128 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:00:59.249 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:00:59.251 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:00:59.261 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":153,"stamp":"2021-02-12T18:00:58.433643+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594392,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163519,"kb_avail":188149288,"statfs":{"total":193273528320,"available":192664870912,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167443508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002083"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"27'6733","reported_seq":"12295","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899186+0000","last_change":"2021-02-12T17:59:52.763903+0000","last_active":"2021-02-12T18:00:34.899186+0000","last_peered":"2021-02-12T18:00:34.899186+0000","last_clean":"2021-02-12T18:00:34.899186+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:00:34.899186+0000","last_undegraded":"2021-02-12T18:00:34.899186+0000","last_fullsized":"2021-02-12T18:00:34.899186+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6733,"ondisk_log_size":6733,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":500822016,"data_stored":500809800,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53762,"ondisk_log_size":53762,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738416,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307212,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":91776,"kb_avail":94064628,"statfs":{"total":96636764160,"available":96322179072,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":93978624},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57499999999999996}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.66300000000000003},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.69699999999999995}]}]},{"osd":0,"up_from":0,"seq":201863462918,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084310,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287180,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":71743,"kb_avail":94084660,"statfs":{"total":96636764160,"available":96342691840,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73464884},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.47199999999999998}]},{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.437},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.63200000000000001}]}]}]}} 2021-02-12T18:00:59.266 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:00:59.266 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:00:59.266 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:00:59.266 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:00:59.266 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:00:59.267 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:00:59.267 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:00:59.267 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:00:59.267 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:01:19.268 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:01:19.573 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:01:19.575 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:01:19.588 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":163,"stamp":"2021-02-12T18:01:18.436395+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53836,"ondisk_log_size":53836,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594520,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163647,"kb_avail":188149160,"statfs":{"total":193273528320,"available":192664739840,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167574580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001657"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'6807","reported_seq":"12371","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:01:13.169943+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:01:13.169943+0000","last_peered":"2021-02-12T18:01:13.169943+0000","last_clean":"2021-02-12T18:01:13.169943+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:01:13.169943+0000","last_undegraded":"2021-02-12T18:01:13.169943+0000","last_fullsized":"2021-02-12T18:01:13.169943+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6807,"ondisk_log_size":6807,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":486076416,"data_stored":486064200,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53836,"ondisk_log_size":53836,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738420,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307276,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":91840,"kb_avail":94064564,"statfs":{"total":96636764160,"available":96322113536,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94044160},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45600000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28299999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.55100000000000005,"5min":0.55100000000000005,"15min":0.55100000000000005},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.311,"5min":0.311,"15min":0.311},"max":{"1min":0.56299999999999994,"5min":0.56299999999999994,"15min":0.56299999999999994},"last":0.58699999999999997}]}]},{"osd":0,"up_from":0,"seq":201863462922,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084314,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287244,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":71807,"kb_avail":94084596,"statfs":{"total":96636764160,"available":96342626304,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73530420},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Feb 12 18:00:24 2021","interfaces":[{"interface":"back","average":{"1min":0.39300000000000002,"5min":0.39300000000000002,"15min":0.39300000000000002},"min":{"1min":0.27300000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.50800000000000001,"5min":0.50800000000000001,"15min":0.50800000000000001},"min":{"1min":0.38500000000000001,"5min":0.38500000000000001,"15min":0.38500000000000001},"max":{"1min":0.755,"5min":0.755,"15min":0.755},"last":0.55200000000000005}]}]}]}} 2021-02-12T18:01:19.588 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:19.589 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:19.589 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:19.589 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:19.589 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:19.589 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:19.589 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:19.589 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:19.589 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:01:39.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:01:39.885 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:01:39.887 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:01:39.897 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":173,"stamp":"2021-02-12T18:01:38.439170+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53911,"ondisk_log_size":53911,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594712,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163839,"kb_avail":188148968,"statfs":{"total":193273528320,"available":192664543232,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167771188},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'6882","reported_seq":"12446","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:01:32.419129+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:01:32.419129+0000","last_peered":"2021-02-12T18:01:32.419129+0000","last_clean":"2021-02-12T18:01:32.419129+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:01:32.419129+0000","last_undegraded":"2021-02-12T18:01:32.419129+0000","last_fullsized":"2021-02-12T18:01:32.419129+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6882,"ondisk_log_size":6882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":471330816,"data_stored":471318600,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53911,"ondisk_log_size":53911,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738424,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307404,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":91968,"kb_avail":94064436,"statfs":{"total":96636764160,"available":96321982464,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94175232},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.46899999999999997}]},{"osd":1,"last update":"Fri Feb 12 18:01:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.41099999999999998,"15min":0.38900000000000001},"min":{"1min":0.40799999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.44,"15min":0.41399999999999998},"min":{"1min":0.36299999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.63700000000000001}]}]},{"osd":0,"up_from":0,"seq":201863462926,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084318,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":287308,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":71871,"kb_avail":94084532,"statfs":{"total":96636764160,"available":96342560768,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":73595956},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39200000000000002}]},{"osd":2,"last update":"Fri Feb 12 18:01:24 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.27700000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78600000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74399999999999999,"5min":0.755,"15min":0.755},"last":0.41599999999999998}]}]}]}} 2021-02-12T18:01:39.898 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:39.898 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:39.898 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:39.898 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:39.898 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:39.898 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:39.899 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:39.899 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:01:39.899 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:01:59.899 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:02:00.200 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:02:00.201 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:02:00.212 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":183,"stamp":"2021-02-12T18:01:58.441846+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53986,"ondisk_log_size":53986,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":598936,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":168063,"kb_avail":188144744,"statfs":{"total":193273528320,"available":192660217856,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":172096564},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001590"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'6957","reported_seq":"12521","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:01:51.679052+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:01:51.679052+0000","last_peered":"2021-02-12T18:01:51.679052+0000","last_clean":"2021-02-12T18:01:51.679052+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:01:51.679052+0000","last_undegraded":"2021-02-12T18:01:51.679052+0000","last_fullsized":"2021-02-12T18:01:51.679052+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6957,"ondisk_log_size":6957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":456708096,"data_stored":456695880,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":53986,"ondisk_log_size":53986,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738428,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":307468,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":92032,"kb_avail":94064372,"statfs":{"total":96636764160,"available":96321916928,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":94240768},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:01:47 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.42499999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:01:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.41099999999999998,"15min":0.38900000000000001},"min":{"1min":0.40799999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.34699999999999998},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.44,"15min":0.41399999999999998},"min":{"1min":0.36299999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.51600000000000001}]}]},{"osd":0,"up_from":0,"seq":201863462930,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084322,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291468,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":76031,"kb_avail":94080372,"statfs":{"total":96636764160,"available":96338300928,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":77855796},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:01:47 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.439},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.51000000000000001}]},{"osd":2,"last update":"Fri Feb 12 18:01:24 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.27700000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78600000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.67600000000000005},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74399999999999999,"5min":0.755,"15min":0.755},"last":0.64900000000000002}]}]}]}} 2021-02-12T18:02:00.213 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:00.213 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:00.213 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:00.213 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:00.213 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:00.213 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:00.214 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:00.214 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:00.214 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:02:20.215 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:02:20.514 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:02:20.516 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:02:20.526 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":194,"stamp":"2021-02-12T18:02:20.444868+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54086,"ondisk_log_size":54086,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603160,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":172287,"kb_avail":188140520,"statfs":{"total":193273528320,"available":192655892480,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":176421940},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001637"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7057","reported_seq":"12621","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:02:17.358921+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:02:17.358921+0000","last_peered":"2021-02-12T18:02:17.358921+0000","last_clean":"2021-02-12T18:02:17.358921+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:02:17.358921+0000","last_undegraded":"2021-02-12T18:02:17.358921+0000","last_fullsized":"2021-02-12T18:02:17.358921+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7057,"ondisk_log_size":7057,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":442576896,"data_stored":442568752,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54086,"ondisk_log_size":54086,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738433,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":311628,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":96192,"kb_avail":94060212,"statfs":{"total":96636764160,"available":96317657088,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":98500608},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:01:47 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.435},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Feb 12 18:01:26 2021","interfaces":[{"interface":"back","average":{"1min":0.54200000000000004,"5min":0.41099999999999998,"15min":0.38900000000000001},"min":{"1min":0.40799999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.70399999999999996,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.44,"15min":0.41399999999999998},"min":{"1min":0.36299999999999999,"5min":0.311,"15min":0.311},"max":{"1min":0.82899999999999996,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.68899999999999995}]}]},{"osd":0,"up_from":0,"seq":201863462934,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084326,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291532,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":76095,"kb_avail":94080308,"statfs":{"total":96636764160,"available":96338235392,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":77921332},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:01:47 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Feb 12 18:01:24 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.40999999999999998,"15min":0.39800000000000002},"min":{"1min":0.27700000000000002,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.78600000000000003,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.42999999999999999},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.52300000000000002,"15min":0.51300000000000001},"min":{"1min":0.38100000000000001,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.74399999999999999,"5min":0.755,"15min":0.755},"last":0.60199999999999998}]}]}]}} 2021-02-12T18:02:20.527 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:20.527 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:20.527 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:20.528 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:20.528 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:20.528 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:20.528 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:20.528 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:20.528 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:02:40.528 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:02:40.827 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:02:40.829 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:02:40.839 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":204,"stamp":"2021-02-12T18:02:40.447464+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54161,"ondisk_log_size":54161,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":586968,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":156095,"kb_avail":188156712,"statfs":{"total":193273528320,"available":192672473088,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":159841332},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001510"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7132","reported_seq":"12696","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:02:36.612627+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:02:36.612627+0000","last_peered":"2021-02-12T18:02:36.612627+0000","last_clean":"2021-02-12T18:02:36.612627+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:02:36.612627+0000","last_undegraded":"2021-02-12T18:02:36.612627+0000","last_fullsized":"2021-02-12T18:02:36.612627+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7132,"ondisk_log_size":7132,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54161,"ondisk_log_size":54161,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738437,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295308,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":79872,"kb_avail":94076532,"statfs":{"total":96636764160,"available":96334368768,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":81788928},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:01:47 2021","interfaces":[{"interface":"back","average":{"1min":0.47099999999999997,"5min":0.47099999999999997,"15min":0.47099999999999997},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41799999999999998},{"interface":"front","average":{"1min":0.54100000000000004,"5min":0.54100000000000004,"15min":0.54100000000000004},"min":{"1min":0.45300000000000001,"5min":0.45300000000000001,"15min":0.45300000000000001},"max":{"1min":0.72999999999999998,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.36299999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:02:26 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.43099999999999999,"15min":0.39600000000000002},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63100000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.441},{"interface":"front","average":{"1min":0.625,"5min":0.48399999999999999,"15min":0.42899999999999999},"min":{"1min":0.45300000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.79800000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57799999999999996}]}]},{"osd":0,"up_from":0,"seq":201863462938,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084330,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291660,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":76223,"kb_avail":94080180,"statfs":{"total":96636764160,"available":96338104320,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":78052404},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:01:47 2021","interfaces":[{"interface":"back","average":{"1min":0.46300000000000002,"5min":0.46300000000000002,"15min":0.46300000000000002},"min":{"1min":0.35199999999999998,"5min":0.35199999999999998,"15min":0.35199999999999998},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.35199999999999998},{"interface":"front","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.39900000000000002,"5min":0.39900000000000002,"15min":0.39900000000000002},"max":{"1min":0.67900000000000005,"5min":0.67900000000000005,"15min":0.67900000000000005},"last":0.42799999999999999}]},{"osd":2,"last update":"Fri Feb 12 18:02:26 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.43099999999999999,"15min":0.40500000000000003},"min":{"1min":0.371,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.39200000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.51800000000000002},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67200000000000004,"5min":0.755,"15min":0.755},"last":0.52800000000000002}]}]}]}} 2021-02-12T18:02:40.840 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:40.840 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:40.840 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:40.841 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:40.841 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:40.841 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:40.841 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:40.841 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:02:40.841 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.7 2021-02-12T18:02:41.138 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.7 on osd.1 to deep-scrub 2021-02-12T18:02:41.148 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.6 2021-02-12T18:02:41.448 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.6 on osd.1 to deep-scrub 2021-02-12T18:02:41.461 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.1 2021-02-12T18:02:41.761 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.1 on osd.2 to deep-scrub 2021-02-12T18:02:41.772 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2021-02-12T18:02:42.070 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2021-02-12T18:02:42.080 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.2 2021-02-12T18:02:42.379 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.2 on osd.1 to deep-scrub 2021-02-12T18:02:42.389 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.3 2021-02-12T18:02:42.689 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.3 on osd.1 to deep-scrub 2021-02-12T18:02:42.700 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.4 2021-02-12T18:02:43.008 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.4 on osd.1 to deep-scrub 2021-02-12T18:02:43.022 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.5 2021-02-12T18:02:43.325 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.5 on osd.2 to deep-scrub 2021-02-12T18:02:43.335 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:03:03.336 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:03:03.635 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:03:03.637 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:03:03.647 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":215,"stamp":"2021-02-12T18:03:02.450232+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54236,"ondisk_log_size":54236,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":587096,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":156223,"kb_avail":188156584,"statfs":{"total":193273528320,"available":192672342016,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":159972404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001463"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7207","reported_seq":"12771","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:02:55.874518+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:02:55.874518+0000","last_peered":"2021-02-12T18:02:55.874518+0000","last_clean":"2021-02-12T18:02:55.874518+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:02:55.874518+0000","last_undegraded":"2021-02-12T18:02:55.874518+0000","last_fullsized":"2021-02-12T18:02:55.874518+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7207,"ondisk_log_size":7207,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54236,"ondisk_log_size":54236,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738441,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295372,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":79936,"kb_avail":94076468,"statfs":{"total":96636764160,"available":96334303232,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":81854464},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:02:48 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.498,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.52300000000000002,"15min":0.53500000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.54500000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.46600000000000003}]},{"osd":1,"last update":"Fri Feb 12 18:02:26 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.43099999999999999,"15min":0.39600000000000002},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63100000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.49199999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.48399999999999999,"15min":0.42899999999999999},"min":{"1min":0.45300000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.79800000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.66500000000000004}]}]},{"osd":0,"up_from":0,"seq":201863462943,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084334,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291724,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":76287,"kb_avail":94080116,"statfs":{"total":96636764160,"available":96338038784,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":78117940},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:02:50 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.55500000000000005},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.69099999999999995}]},{"osd":2,"last update":"Fri Feb 12 18:02:26 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.43099999999999999,"15min":0.40500000000000003},"min":{"1min":0.371,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.46800000000000003},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.51800000000000002},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67200000000000004,"5min":0.755,"15min":0.755},"last":0.51600000000000001}]}]}]}} 2021-02-12T18:03:03.648 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:03.648 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:03.649 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:03.649 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:03.649 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:03.649 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:03.649 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:03.649 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:03.649 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:03:23.651 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:03:23.944 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:03:23.946 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:03:23.956 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":225,"stamp":"2021-02-12T18:03:22.452797+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54311,"ondisk_log_size":54311,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":587288,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":156415,"kb_avail":188156392,"statfs":{"total":193273528320,"available":192672145408,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":160169012},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7282","reported_seq":"12846","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:03:15.135129+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:03:15.135129+0000","last_peered":"2021-02-12T18:03:15.135129+0000","last_clean":"2021-02-12T18:03:15.135129+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:03:15.135129+0000","last_undegraded":"2021-02-12T18:03:15.135129+0000","last_fullsized":"2021-02-12T18:03:15.135129+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7282,"ondisk_log_size":7282,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54311,"ondisk_log_size":54311,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738445,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":295500,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":80064,"kb_avail":94076340,"statfs":{"total":96636764160,"available":96334172160,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":81985536},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:02:48 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.498,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41599999999999998},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.52300000000000002,"15min":0.53500000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.54500000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.45900000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:02:26 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.43099999999999999,"15min":0.39600000000000002},"min":{"1min":0.34699999999999998,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.63100000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.625,"5min":0.48399999999999999,"15min":0.42899999999999999},"min":{"1min":0.45300000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.79800000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.56599999999999995}]}]},{"osd":0,"up_from":0,"seq":201863462947,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084338,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291788,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":76351,"kb_avail":94080052,"statfs":{"total":96636764160,"available":96337973248,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":78183476},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:02:50 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.378},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.48699999999999999}]},{"osd":2,"last update":"Fri Feb 12 18:02:26 2021","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.43099999999999999,"15min":0.40500000000000003},"min":{"1min":0.371,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.67600000000000005,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.51900000000000002},{"interface":"front","average":{"1min":0.58899999999999997,"5min":0.54000000000000004,"15min":0.51800000000000002},"min":{"1min":0.41599999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.67200000000000004,"5min":0.755,"15min":0.755},"last":0.53600000000000003}]}]}]}} 2021-02-12T18:03:23.957 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:23.957 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:23.957 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:23.958 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:23.958 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:23.958 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:23.958 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:23.958 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:23.958 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:03:43.959 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:03:44.256 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:03:44.257 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:03:44.268 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":235,"stamp":"2021-02-12T18:03:42.455447+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54386,"ondisk_log_size":54386,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":591512,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":160639,"kb_avail":188152168,"statfs":{"total":193273528320,"available":192667820032,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":164494388},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":6,"apply_latency_ms":6,"commit_latency_ns":6000000,"apply_latency_ns":6000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001562"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7357","reported_seq":"12921","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:03:34.383579+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:03:34.383579+0000","last_peered":"2021-02-12T18:03:34.383579+0000","last_clean":"2021-02-12T18:03:34.383579+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:03:34.383579+0000","last_undegraded":"2021-02-12T18:03:34.383579+0000","last_fullsized":"2021-02-12T18:03:34.383579+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7357,"ondisk_log_size":7357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54386,"ondisk_log_size":54386,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738449,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299660,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":84224,"kb_avail":94072180,"statfs":{"total":96636764160,"available":96329912320,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86245376},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:02:48 2021","interfaces":[{"interface":"back","average":{"1min":0.436,"5min":0.46400000000000002,"15min":0.46800000000000003},"min":{"1min":0.35499999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.498,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.45100000000000001,"5min":0.52300000000000002,"15min":0.53500000000000003},"min":{"1min":0.36299999999999999,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.54500000000000004,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.51500000000000001}]},{"osd":1,"last update":"Fri Feb 12 18:03:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45000000000000001,"15min":0.40200000000000002},"min":{"1min":0.40699999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.57299999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.47799999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.52500000000000002,"15min":0.443},"min":{"1min":0.56599999999999995,"5min":0.311,"15min":0.311},"max":{"1min":0.67300000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.626}]}]},{"osd":0,"up_from":0,"seq":201863462951,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084342,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291852,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":76415,"kb_avail":94079988,"statfs":{"total":96636764160,"available":96337907712,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":78249012},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:02:50 2021","interfaces":[{"interface":"back","average":{"1min":0.45600000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.67700000000000005,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.60899999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.80400000000000005,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.51700000000000002}]},{"osd":2,"last update":"Fri Feb 12 18:03:28 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44600000000000001,"15min":0.40999999999999998},"min":{"1min":0.373,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59499999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.54300000000000004},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54200000000000004,"15min":0.51900000000000002},"min":{"1min":0.41299999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.622,"5min":0.755,"15min":0.755},"last":0.498}]}]}]}} 2021-02-12T18:03:44.269 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:44.269 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:44.269 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:44.269 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:44.269 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:44.269 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:44.270 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:44.270 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:03:44.270 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:04:04.271 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:04:04.576 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:04:04.577 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:04:04.588 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":246,"stamp":"2021-02-12T18:04:04.458307+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54486,"ondisk_log_size":54486,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":591704,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":160831,"kb_avail":188151976,"statfs":{"total":193273528320,"available":192667623424,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":164690996},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001557"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7457","reported_seq":"13021","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:04:00.068388+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:04:00.068388+0000","last_peered":"2021-02-12T18:04:00.068388+0000","last_clean":"2021-02-12T18:04:00.068388+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:04:00.068388+0000","last_undegraded":"2021-02-12T18:04:00.068388+0000","last_fullsized":"2021-02-12T18:04:00.068388+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7457,"ondisk_log_size":7457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54486,"ondisk_log_size":54486,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738454,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299724,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":84288,"kb_avail":94072116,"statfs":{"total":96636764160,"available":96329846784,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86310912},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.45600000000000002,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.51800000000000002,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.40899999999999997},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.38800000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.53700000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.58399999999999996}]},{"osd":1,"last update":"Fri Feb 12 18:03:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45000000000000001,"15min":0.40200000000000002},"min":{"1min":0.40699999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.57299999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.60599999999999998},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.52500000000000002,"15min":0.443},"min":{"1min":0.56599999999999995,"5min":0.311,"15min":0.311},"max":{"1min":0.67300000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.46100000000000002}]}]},{"osd":0,"up_from":0,"seq":201863462955,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084347,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":291980,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":76543,"kb_avail":94079860,"statfs":{"total":96636764160,"available":96337776640,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":78380084},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:03:54 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.378,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.60899999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.40000000000000002},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69099999999999995,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.56399999999999995}]},{"osd":2,"last update":"Fri Feb 12 18:03:28 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44600000000000001,"15min":0.40999999999999998},"min":{"1min":0.373,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59499999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.53600000000000003},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54200000000000004,"15min":0.51900000000000002},"min":{"1min":0.41299999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.622,"5min":0.755,"15min":0.755},"last":0.502}]}]}]}} 2021-02-12T18:04:04.589 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:04.589 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:04.589 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:04.589 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:04.589 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:04.590 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:04.590 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:04.590 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:04.590 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:04:24.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:04:24.887 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:04:24.889 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:04:24.900 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":256,"stamp":"2021-02-12T18:04:24.460974+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54561,"ondisk_log_size":54561,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595928,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":165055,"kb_avail":188147752,"statfs":{"total":193273528320,"available":192663298048,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":169016372},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001517"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7532","reported_seq":"13096","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:04:19.324831+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:04:19.324831+0000","last_peered":"2021-02-12T18:04:19.324831+0000","last_clean":"2021-02-12T18:04:19.324831+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:04:19.324831+0000","last_undegraded":"2021-02-12T18:04:19.324831+0000","last_fullsized":"2021-02-12T18:04:19.324831+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7532,"ondisk_log_size":7532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54561,"ondisk_log_size":54561,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738458,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":299788,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":84352,"kb_avail":94072052,"statfs":{"total":96636764160,"available":96329781248,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":86376448},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.45600000000000002,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.51800000000000002,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.44900000000000001},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.38800000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.53700000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.496}]},{"osd":1,"last update":"Fri Feb 12 18:03:29 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.45000000000000001,"15min":0.40200000000000002},"min":{"1min":0.40699999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.57299999999999995,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.52500000000000002,"15min":0.443},"min":{"1min":0.56599999999999995,"5min":0.311,"15min":0.311},"max":{"1min":0.67300000000000004,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.64400000000000002}]}]},{"osd":0,"up_from":0,"seq":201863462959,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084351,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296140,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":80703,"kb_avail":94075700,"statfs":{"total":96636764160,"available":96333516800,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":82639924},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:03:54 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.378,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.60899999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69099999999999995,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.47699999999999998}]},{"osd":2,"last update":"Fri Feb 12 18:03:28 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.44600000000000001,"15min":0.40999999999999998},"min":{"1min":0.373,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.59499999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.54200000000000004,"15min":0.51900000000000002},"min":{"1min":0.41299999999999998,"5min":0.38100000000000001,"15min":0.38100000000000001},"max":{"1min":0.622,"5min":0.755,"15min":0.755},"last":0.501}]}]}]}} 2021-02-12T18:04:24.901 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:24.901 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:24.901 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:24.901 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:24.901 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:24.901 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:24.901 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:24.901 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:24.902 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:04:44.903 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:04:45.231 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:04:45.232 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:04:45.243 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":266,"stamp":"2021-02-12T18:04:44.463770+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54636,"ondisk_log_size":54636,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600216,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":169343,"kb_avail":188143464,"statfs":{"total":193273528320,"available":192658907136,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173407284},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001660"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7607","reported_seq":"13171","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:04:38.584923+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:04:38.584923+0000","last_peered":"2021-02-12T18:04:38.584923+0000","last_clean":"2021-02-12T18:04:38.584923+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:04:38.584923+0000","last_undegraded":"2021-02-12T18:04:38.584923+0000","last_fullsized":"2021-02-12T18:04:38.584923+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7607,"ondisk_log_size":7607,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54636,"ondisk_log_size":54636,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738462,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304012,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":88576,"kb_avail":94067828,"statfs":{"total":96636764160,"available":96325455872,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90701824},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:03:53 2021","interfaces":[{"interface":"back","average":{"1min":0.433,"5min":0.45600000000000002,"15min":0.46600000000000003},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.51800000000000002,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.46600000000000003,"5min":0.50800000000000001,"15min":0.53000000000000003},"min":{"1min":0.38800000000000001,"5min":0.36299999999999999,"15min":0.36299999999999999},"max":{"1min":0.53700000000000003,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.5}]},{"osd":1,"last update":"Fri Feb 12 18:04:32 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.40899999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60599999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.56699999999999995,"15min":0.45700000000000002},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.75600000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.60699999999999998}]}]},{"osd":0,"up_from":0,"seq":201863462963,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084355,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296204,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":80767,"kb_avail":94075636,"statfs":{"total":96636764160,"available":96333451264,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":82705460},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:03:54 2021","interfaces":[{"interface":"back","average":{"1min":0.46200000000000002,"5min":0.46100000000000002,"15min":0.46200000000000002},"min":{"1min":0.378,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.60899999999999999,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.51200000000000001},{"interface":"front","average":{"1min":0.51800000000000002,"5min":0.52200000000000002,"15min":0.52000000000000002},"min":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.69099999999999995,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.58299999999999996}]},{"osd":2,"last update":"Fri Feb 12 18:04:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46600000000000003,"15min":0.41699999999999998},"min":{"1min":0.38800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.46300000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54100000000000004,"15min":0.51900000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.63100000000000001,"5min":0.755,"15min":0.755},"last":0.436}]}]}]}} 2021-02-12T18:04:45.244 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:45.244 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:45.244 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:45.244 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:45.244 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:45.244 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:45.244 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:45.245 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:04:45.245 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.7 2021-02-12T18:04:45.542 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.7 on osd.1 to deep-scrub 2021-02-12T18:04:45.552 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.6 2021-02-12T18:04:45.853 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.6 on osd.1 to deep-scrub 2021-02-12T18:04:45.863 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.1 2021-02-12T18:04:46.161 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.1 on osd.2 to deep-scrub 2021-02-12T18:04:46.171 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2021-02-12T18:04:46.471 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2021-02-12T18:04:46.482 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.2 2021-02-12T18:04:46.783 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.2 on osd.1 to deep-scrub 2021-02-12T18:04:46.794 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.3 2021-02-12T18:04:47.093 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.3 on osd.1 to deep-scrub 2021-02-12T18:04:47.104 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.4 2021-02-12T18:04:47.408 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.4 on osd.1 to deep-scrub 2021-02-12T18:04:47.418 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.5 2021-02-12T18:04:47.719 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.5 on osd.2 to deep-scrub 2021-02-12T18:04:47.729 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:05:07.730 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:05:08.024 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:05:08.026 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:05:08.036 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":277,"stamp":"2021-02-12T18:05:06.466626+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54711,"ondisk_log_size":54711,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600344,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":169471,"kb_avail":188143336,"statfs":{"total":193273528320,"available":192658776064,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173538356},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001538"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7682","reported_seq":"13246","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:04:57.840410+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:04:57.840410+0000","last_peered":"2021-02-12T18:04:57.840410+0000","last_clean":"2021-02-12T18:04:57.840410+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:04:57.840410+0000","last_undegraded":"2021-02-12T18:04:57.840410+0000","last_fullsized":"2021-02-12T18:04:57.840410+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7682,"ondisk_log_size":7682,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54711,"ondisk_log_size":54711,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738466,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304076,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":88640,"kb_avail":94067764,"statfs":{"total":96636764160,"available":96325390336,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90767360},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:04:53 2021","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.44600000000000001,"15min":0.46200000000000002},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49199999999999999,"15min":0.52400000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.58399999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40999999999999998}]},{"osd":1,"last update":"Fri Feb 12 18:04:32 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.40899999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60599999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.56699999999999995,"15min":0.45700000000000002},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.75600000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.628}]}]},{"osd":0,"up_from":0,"seq":201863462967,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084359,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296268,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":80831,"kb_avail":94075572,"statfs":{"total":96636764160,"available":96333385728,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":82770996},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:04:57 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.56999999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.39200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.64600000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.55400000000000005}]},{"osd":2,"last update":"Fri Feb 12 18:04:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46600000000000003,"15min":0.41699999999999998},"min":{"1min":0.38800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.52300000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54100000000000004,"15min":0.51900000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.63100000000000001,"5min":0.755,"15min":0.755},"last":0.57399999999999995}]}]}]}} 2021-02-12T18:05:08.037 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:08.037 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:08.037 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:08.037 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:08.038 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:08.038 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:08.038 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:08.038 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:08.038 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:05:28.039 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:05:29.164 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:05:29.166 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:05:29.176 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":288,"stamp":"2021-02-12T18:05:28.469389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54811,"ondisk_log_size":54811,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600536,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":169663,"kb_avail":188143144,"statfs":{"total":193273528320,"available":192658579456,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173734964},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001504"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7782","reported_seq":"13346","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:05:23.525438+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:05:23.525438+0000","last_peered":"2021-02-12T18:05:23.525438+0000","last_clean":"2021-02-12T18:05:23.525438+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:05:23.525438+0000","last_undegraded":"2021-02-12T18:05:23.525438+0000","last_fullsized":"2021-02-12T18:05:23.525438+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7782,"ondisk_log_size":7782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54811,"ondisk_log_size":54811,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738470,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304140,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":88704,"kb_avail":94067700,"statfs":{"total":96636764160,"available":96325324800,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90832896},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:04:53 2021","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.44600000000000001,"15min":0.46200000000000002},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49199999999999999,"15min":0.52400000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.58399999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.40999999999999998}]},{"osd":1,"last update":"Fri Feb 12 18:04:32 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.46999999999999997,"15min":0.40899999999999997},"min":{"1min":0.40899999999999997,"5min":0.28299999999999997,"15min":0.28299999999999997},"max":{"1min":0.60599999999999998,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.438},{"interface":"front","average":{"1min":0.61299999999999999,"5min":0.56699999999999995,"15min":0.45700000000000002},"min":{"1min":0.39000000000000001,"5min":0.311,"15min":0.311},"max":{"1min":0.75600000000000001,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.55800000000000005}]}]},{"osd":0,"up_from":0,"seq":201863462972,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084364,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296396,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":80959,"kb_avail":94075444,"statfs":{"total":96636764160,"available":96333254656,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":82902068},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:04:57 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.56999999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.48499999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.39200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.64600000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.621}]},{"osd":2,"last update":"Fri Feb 12 18:04:29 2021","interfaces":[{"interface":"back","average":{"1min":0.48899999999999999,"5min":0.46600000000000003,"15min":0.41699999999999998},"min":{"1min":0.38800000000000001,"5min":0.27300000000000002,"15min":0.27300000000000002},"max":{"1min":0.58999999999999997,"5min":0.78900000000000003,"15min":0.78900000000000003},"last":0.44500000000000001},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.54100000000000004,"15min":0.51900000000000002},"min":{"1min":0.36699999999999999,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.63100000000000001,"5min":0.755,"15min":0.755},"last":0.51200000000000001}]}]}]}} 2021-02-12T18:05:29.177 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:29.177 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:29.177 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:29.177 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:29.177 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:29.178 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:29.178 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:29.178 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:29.178 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:05:49.179 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:05:49.475 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:05:49.477 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:05:49.488 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":298,"stamp":"2021-02-12T18:05:48.472086+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54886,"ondisk_log_size":54886,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600664,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":169791,"kb_avail":188143016,"statfs":{"total":193273528320,"available":192658448384,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173866036},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001668"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7857","reported_seq":"13421","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:05:42.779388+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:05:42.779388+0000","last_peered":"2021-02-12T18:05:42.779388+0000","last_clean":"2021-02-12T18:05:42.779388+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:05:42.779388+0000","last_undegraded":"2021-02-12T18:05:42.779388+0000","last_fullsized":"2021-02-12T18:05:42.779388+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7857,"ondisk_log_size":7857,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54886,"ondisk_log_size":54886,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738474,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":304204,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":88768,"kb_avail":94067636,"statfs":{"total":96636764160,"available":96325259264,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":90898432},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:04:53 2021","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.44600000000000001,"15min":0.46200000000000002},"min":{"1min":0.35799999999999998,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.51700000000000002,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.46200000000000002,"5min":0.49199999999999999,"15min":0.52400000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.58399999999999996,"5min":0.72999999999999998,"15min":0.72999999999999998},"last":0.53400000000000003}]},{"osd":1,"last update":"Fri Feb 12 18:05:33 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.48399999999999999,"15min":0.41399999999999998},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.28299999999999997},"max":{"1min":0.55200000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.5},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.46899999999999997},"min":{"1min":0.51200000000000001,"5min":0.36299999999999999,"15min":0.311},"max":{"1min":0.66400000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.65300000000000002}]}]},{"osd":0,"up_from":0,"seq":201863462976,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084368,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296460,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":81023,"kb_avail":94075380,"statfs":{"total":96636764160,"available":96333189120,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":82967604},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:04:57 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45600000000000002,"15min":0.46000000000000002},"min":{"1min":0.35199999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"max":{"1min":0.56999999999999995,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.58499999999999996},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52300000000000002,"15min":0.52000000000000002},"min":{"1min":0.39200000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.64600000000000002,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.45500000000000002}]},{"osd":2,"last update":"Fri Feb 12 18:05:33 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.42299999999999999},"min":{"1min":0.42999999999999999,"5min":0.27700000000000002,"15min":0.27300000000000002},"max":{"1min":0.61299999999999999,"5min":0.78600000000000003,"15min":0.78900000000000003},"last":0.50600000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53500000000000003,"15min":0.51700000000000002},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.58899999999999997,"5min":0.74399999999999999,"15min":0.755},"last":0.48199999999999998}]}]}]}} 2021-02-12T18:05:49.488 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:49.489 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:49.489 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:49.489 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:49.489 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:49.489 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:49.489 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:49.489 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:05:49.489 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:06:09.490 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:06:09.799 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:06:09.801 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:06:09.811 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":308,"stamp":"2021-02-12T18:06:08.474754+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54961,"ondisk_log_size":54961,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604888,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":174015,"kb_avail":188138792,"statfs":{"total":193273528320,"available":192654123008,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":178191412},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001566"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'7932","reported_seq":"13496","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:06:02.044876+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:06:02.044876+0000","last_peered":"2021-02-12T18:06:02.044876+0000","last_clean":"2021-02-12T18:06:02.044876+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:06:02.044876+0000","last_undegraded":"2021-02-12T18:06:02.044876+0000","last_fullsized":"2021-02-12T18:06:02.044876+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":7932,"ondisk_log_size":7932,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":54961,"ondisk_log_size":54961,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738478,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308364,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":92928,"kb_avail":94063476,"statfs":{"total":96636764160,"available":96320999424,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95158272},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:05:54 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44,"15min":0.46000000000000002},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.53300000000000003,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.40699999999999997},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.47699999999999998,"15min":0.51900000000000002},"min":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45900000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:05:33 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.48399999999999999,"15min":0.41399999999999998},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.28299999999999997},"max":{"1min":0.55200000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.46899999999999997},"min":{"1min":0.51200000000000001,"5min":0.36299999999999999,"15min":0.311},"max":{"1min":0.66400000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.67100000000000004}]}]},{"osd":0,"up_from":0,"seq":201863462980,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084372,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296524,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":81087,"kb_avail":94075316,"statfs":{"total":96636764160,"available":96333123584,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":83033140},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:05:59 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45100000000000001,"15min":0.45900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.58499999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.54500000000000004},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.43099999999999999}]},{"osd":2,"last update":"Fri Feb 12 18:05:33 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.42299999999999999},"min":{"1min":0.42999999999999999,"5min":0.27700000000000002,"15min":0.27300000000000002},"max":{"1min":0.61299999999999999,"5min":0.78600000000000003,"15min":0.78900000000000003},"last":0.502},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53500000000000003,"15min":0.51700000000000002},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.58899999999999997,"5min":0.74399999999999999,"15min":0.755},"last":0.57199999999999995}]}]}]}} 2021-02-12T18:06:09.813 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:09.813 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:09.813 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:09.813 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:09.813 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:09.813 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:09.813 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:09.814 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:09.814 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:06:29.815 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:06:30.115 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:06:30.117 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:06:30.128 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":318,"stamp":"2021-02-12T18:06:28.477214+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55061,"ondisk_log_size":55061,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":609176,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":178303,"kb_avail":188134504,"statfs":{"total":193273528320,"available":192649732096,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":182582324},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001406"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8032","reported_seq":"13596","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:06:27.717992+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:06:27.717992+0000","last_peered":"2021-02-12T18:06:27.717992+0000","last_clean":"2021-02-12T18:06:27.717992+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:06:27.717992+0000","last_undegraded":"2021-02-12T18:06:27.717992+0000","last_fullsized":"2021-02-12T18:06:27.717992+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8032,"ondisk_log_size":8032,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55061,"ondisk_log_size":55061,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738482,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308492,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":93056,"kb_avail":94063348,"statfs":{"total":96636764160,"available":96320868352,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95289344},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:05:54 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44,"15min":0.46000000000000002},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.53300000000000003,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.38300000000000001},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.47699999999999998,"15min":0.51900000000000002},"min":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46000000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:05:33 2021","interfaces":[{"interface":"back","average":{"1min":0.45000000000000001,"5min":0.48399999999999999,"15min":0.41399999999999998},"min":{"1min":0.39200000000000002,"5min":0.34699999999999998,"15min":0.28299999999999997},"max":{"1min":0.55200000000000005,"5min":0.70399999999999996,"15min":0.70399999999999996},"last":0.495},{"interface":"front","average":{"1min":0.59199999999999997,"5min":0.60499999999999998,"15min":0.46899999999999997},"min":{"1min":0.51200000000000001,"5min":0.36299999999999999,"15min":0.311},"max":{"1min":0.66400000000000003,"5min":0.82899999999999996,"15min":0.82899999999999996},"last":0.57199999999999995}]}]},{"osd":0,"up_from":0,"seq":201863462984,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084376,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300684,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":85247,"kb_avail":94071156,"statfs":{"total":96636764160,"available":96328863744,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":87292980},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:05:59 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45100000000000001,"15min":0.45900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.58499999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.44400000000000001},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.58999999999999997}]},{"osd":2,"last update":"Fri Feb 12 18:05:33 2021","interfaces":[{"interface":"back","average":{"1min":0.48199999999999998,"5min":0.48299999999999998,"15min":0.42299999999999999},"min":{"1min":0.42999999999999999,"5min":0.27700000000000002,"15min":0.27300000000000002},"max":{"1min":0.61299999999999999,"5min":0.78600000000000003,"15min":0.78900000000000003},"last":0.50700000000000001},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.53500000000000003,"15min":0.51700000000000002},"min":{"1min":0.38200000000000001,"5min":0.36699999999999999,"15min":0.36699999999999999},"max":{"1min":0.58899999999999997,"5min":0.74399999999999999,"15min":0.755},"last":0.46600000000000003}]}]}]}} 2021-02-12T18:06:30.128 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:30.129 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:30.129 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:30.129 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:30.129 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:30.129 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:30.129 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:30.129 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:30.129 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:06:50.131 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:06:50.431 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:06:50.433 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:06:50.443 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":328,"stamp":"2021-02-12T18:06:48.479748+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55136,"ondisk_log_size":55136,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":609368,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":178495,"kb_avail":188134312,"statfs":{"total":193273528320,"available":192649535488,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":182778932},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001522"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8107","reported_seq":"13671","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:06:46.979849+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:06:46.979849+0000","last_peered":"2021-02-12T18:06:46.979849+0000","last_clean":"2021-02-12T18:06:46.979849+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:06:46.979849+0000","last_undegraded":"2021-02-12T18:06:46.979849+0000","last_fullsized":"2021-02-12T18:06:46.979849+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8107,"ondisk_log_size":8107,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55136,"ondisk_log_size":55136,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738486,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":308556,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":93120,"kb_avail":94063284,"statfs":{"total":96636764160,"available":96320802816,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":95354880},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:05:54 2021","interfaces":[{"interface":"back","average":{"1min":0.438,"5min":0.44,"15min":0.46000000000000002},"min":{"1min":0.36299999999999999,"5min":0.35499999999999998,"15min":0.35499999999999998},"max":{"1min":0.53300000000000003,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.46700000000000003,"5min":0.47699999999999998,"15min":0.51900000000000002},"min":{"1min":0.36799999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.79300000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.45200000000000001}]},{"osd":1,"last update":"Fri Feb 12 18:06:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47099999999999997,"15min":0.41999999999999998},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.70399999999999996},"last":0.39800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60999999999999999,"15min":0.48399999999999999},"min":{"1min":0.52700000000000002,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.82899999999999996},"last":0.59199999999999997}]}]},{"osd":0,"up_from":0,"seq":201863462989,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084380,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300812,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":85375,"kb_avail":94071028,"statfs":{"total":96636764160,"available":96328732672,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":87424052},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:05:59 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.45100000000000001,"15min":0.45900000000000002},"min":{"1min":0.32600000000000001,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.58499999999999996,"5min":0.67700000000000005,"15min":0.67700000000000005},"last":0.52000000000000002},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.52800000000000002,"15min":0.52200000000000002},"min":{"1min":0.39800000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66200000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.61599999999999999}]},{"osd":2,"last update":"Fri Feb 12 18:06:37 2021","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48899999999999999,"15min":0.42999999999999999},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.27300000000000002},"max":{"1min":0.63800000000000001,"5min":0.67600000000000005,"15min":0.78900000000000003},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51900000000000002,"15min":0.51600000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.61099999999999999,"5min":0.67200000000000004,"15min":0.755},"last":0.495}]}]}]}} 2021-02-12T18:06:50.444 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:50.445 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:50.445 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:50.445 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:50.445 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:50.445 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:50.445 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:50.445 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:06:50.445 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.7 2021-02-12T18:06:50.747 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.7 on osd.1 to deep-scrub 2021-02-12T18:06:50.758 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.6 2021-02-12T18:06:51.884 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.6 on osd.1 to deep-scrub 2021-02-12T18:06:51.895 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.1 2021-02-12T18:06:52.196 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.1 on osd.2 to deep-scrub 2021-02-12T18:06:52.206 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2021-02-12T18:06:52.506 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2021-02-12T18:06:52.518 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.2 2021-02-12T18:06:52.823 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.2 on osd.1 to deep-scrub 2021-02-12T18:06:52.834 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.3 2021-02-12T18:06:53.135 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.3 on osd.1 to deep-scrub 2021-02-12T18:06:53.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.4 2021-02-12T18:06:53.467 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.4 on osd.1 to deep-scrub 2021-02-12T18:06:53.477 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.5 2021-02-12T18:06:53.777 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.5 on osd.2 to deep-scrub 2021-02-12T18:06:53.787 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:07:13.788 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:07:14.085 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:07:14.087 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:07:14.098 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":340,"stamp":"2021-02-12T18:07:12.482778+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55211,"ondisk_log_size":55211,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":597208,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":166335,"kb_avail":188146472,"statfs":{"total":193273528320,"available":192661987328,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":170327092},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001521"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8182","reported_seq":"13746","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:07:06.234031+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:07:06.234031+0000","last_peered":"2021-02-12T18:07:06.234031+0000","last_clean":"2021-02-12T18:07:06.234031+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:07:06.234031+0000","last_undegraded":"2021-02-12T18:07:06.234031+0000","last_fullsized":"2021-02-12T18:07:06.234031+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8182,"ondisk_log_size":8182,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55211,"ondisk_log_size":55211,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738491,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":312716,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":97280,"kb_avail":94059124,"statfs":{"total":96636764160,"available":96316542976,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":99614720},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:07:00 2021","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.43099999999999999,"15min":0.45700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.53500000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.47399999999999998}]},{"osd":1,"last update":"Fri Feb 12 18:06:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47099999999999997,"15min":0.41999999999999998},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.70399999999999996},"last":0.434},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60999999999999999,"15min":0.48399999999999999},"min":{"1min":0.52700000000000002,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.82899999999999996},"last":0.55500000000000005}]}]},{"osd":0,"up_from":0,"seq":201863462994,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084384,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284492,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":69055,"kb_avail":94087348,"statfs":{"total":96636764160,"available":96345444352,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70712372},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:07:03 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.45800000000000002},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.41999999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.52500000000000002},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66400000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.60599999999999998}]},{"osd":2,"last update":"Fri Feb 12 18:06:37 2021","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48899999999999999,"15min":0.42999999999999999},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.27300000000000002},"max":{"1min":0.63800000000000001,"5min":0.67600000000000005,"15min":0.78900000000000003},"last":0.56999999999999995},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51900000000000002,"15min":0.51600000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.61099999999999999,"5min":0.67200000000000004,"15min":0.755},"last":0.45300000000000001}]}]}]}} 2021-02-12T18:07:14.099 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:14.099 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:14.099 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:14.099 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:14.099 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:14.099 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:14.099 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:14.099 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:14.100 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:07:34.100 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:07:34.393 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:07:34.395 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:07:34.405 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":350,"stamp":"2021-02-12T18:07:32.485288+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55286,"ondisk_log_size":55286,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":580952,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":150079,"kb_avail":188162728,"statfs":{"total":193273528320,"available":192678633472,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":153680948},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001453"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8257","reported_seq":"13821","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:07:25.498101+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:07:25.498101+0000","last_peered":"2021-02-12T18:07:25.498101+0000","last_clean":"2021-02-12T18:07:25.498101+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:07:25.498101+0000","last_undegraded":"2021-02-12T18:07:25.498101+0000","last_fullsized":"2021-02-12T18:07:25.498101+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8257,"ondisk_log_size":8257,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55286,"ondisk_log_size":55286,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738495,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296396,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":80960,"kb_avail":94075444,"statfs":{"total":96636764160,"available":96333254656,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":82903040},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:07:00 2021","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.43099999999999999,"15min":0.45700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.53500000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.49399999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:06:37 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.47099999999999997,"15min":0.41999999999999998},"min":{"1min":0.35499999999999998,"5min":0.34699999999999998,"15min":0.28299999999999997},"max":{"1min":0.65300000000000002,"5min":0.65300000000000002,"15min":0.70399999999999996},"last":0.51800000000000002},{"interface":"front","average":{"1min":0.61399999999999999,"5min":0.60999999999999999,"15min":0.48399999999999999},"min":{"1min":0.52700000000000002,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.71899999999999997,"5min":0.79800000000000004,"15min":0.82899999999999996},"last":0.63300000000000001}]}]},{"osd":0,"up_from":0,"seq":201863462998,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084388,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284556,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":69119,"kb_avail":94087284,"statfs":{"total":96636764160,"available":96345378816,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70777908},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:07:03 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.45800000000000002},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.52500000000000002},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66400000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.63800000000000001}]},{"osd":2,"last update":"Fri Feb 12 18:06:37 2021","interfaces":[{"interface":"back","average":{"1min":0.50700000000000001,"5min":0.48899999999999999,"15min":0.42999999999999999},"min":{"1min":0.42999999999999999,"5min":0.371,"15min":0.27300000000000002},"max":{"1min":0.63800000000000001,"5min":0.67600000000000005,"15min":0.78900000000000003},"last":0.503},{"interface":"front","average":{"1min":0.50600000000000001,"5min":0.51900000000000002,"15min":0.51600000000000001},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.61099999999999999,"5min":0.67200000000000004,"15min":0.755},"last":0.53200000000000003}]}]}]}} 2021-02-12T18:07:34.406 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:34.406 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:34.406 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:34.406 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:34.406 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:34.407 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:34.407 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:34.407 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:34.407 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:07:54.408 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:07:54.717 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:07:54.719 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:07:54.729 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":361,"stamp":"2021-02-12T18:07:54.488240+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55386,"ondisk_log_size":55386,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":581144,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":150271,"kb_avail":188162536,"statfs":{"total":193273528320,"available":192678436864,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":153877556},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001581"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8357","reported_seq":"13921","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:07:51.172440+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:07:51.172440+0000","last_peered":"2021-02-12T18:07:51.172440+0000","last_clean":"2021-02-12T18:07:51.172440+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:07:51.172440+0000","last_undegraded":"2021-02-12T18:07:51.172440+0000","last_fullsized":"2021-02-12T18:07:51.172440+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8357,"ondisk_log_size":8357,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55386,"ondisk_log_size":55386,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738500,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":296524,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":81088,"kb_avail":94075316,"statfs":{"total":96636764160,"available":96333123584,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":83034112},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:07:00 2021","interfaces":[{"interface":"back","average":{"1min":0.42799999999999999,"5min":0.43099999999999999,"15min":0.45700000000000002},"min":{"1min":0.33900000000000002,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.67200000000000004,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.45700000000000002,"5min":0.46000000000000002,"15min":0.51400000000000001},"min":{"1min":0.39100000000000001,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.53500000000000003,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.52700000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:07:41 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42599999999999999},"min":{"1min":0.377,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.70399999999999996},"last":0.501},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.497},"min":{"1min":0.53600000000000003,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.68600000000000005,"5min":0.75600000000000001,"15min":0.82899999999999996},"last":0.84899999999999998}]}]},{"osd":0,"up_from":0,"seq":201863463002,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084393,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284620,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":69183,"kb_avail":94087220,"statfs":{"total":96636764160,"available":96345313280,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70843444},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:07:03 2021","interfaces":[{"interface":"back","average":{"1min":0.45400000000000001,"5min":0.44900000000000001,"15min":0.45800000000000002},"min":{"1min":0.36899999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.70199999999999996,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.41099999999999998},{"interface":"front","average":{"1min":0.56799999999999995,"5min":0.53800000000000003,"15min":0.52500000000000002},"min":{"1min":0.43099999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66400000000000003,"5min":0.80400000000000005,"15min":0.80400000000000005},"last":0.58399999999999996}]},{"osd":2,"last update":"Fri Feb 12 18:07:39 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.436},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.60099999999999998,"5min":0.63800000000000001,"15min":0.78900000000000003},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.755},"last":0.437}]}]}]}} 2021-02-12T18:07:54.730 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:54.731 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:54.731 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:54.731 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:54.731 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:54.731 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:54.731 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:54.731 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:07:54.731 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:08:14.733 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:08:15.031 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:08:15.032 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:08:15.042 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":371,"stamp":"2021-02-12T18:08:14.490862+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55461,"ondisk_log_size":55461,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":585432,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":154559,"kb_avail":188158248,"statfs":{"total":193273528320,"available":192674045952,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":158268468},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001587"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8432","reported_seq":"13996","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:08:10.423167+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:08:10.423167+0000","last_peered":"2021-02-12T18:08:10.423167+0000","last_clean":"2021-02-12T18:08:10.423167+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:08:10.423167+0000","last_undegraded":"2021-02-12T18:08:10.423167+0000","last_fullsized":"2021-02-12T18:08:10.423167+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8432,"ondisk_log_size":8432,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55461,"ondisk_log_size":55461,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738504,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300684,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":85248,"kb_avail":94071156,"statfs":{"total":96636764160,"available":96328863744,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":87293952},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:08:00 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.43099999999999999,"15min":0.45500000000000002},"min":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.51500000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.371},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.35999999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48799999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:07:41 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42599999999999999},"min":{"1min":0.377,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.70399999999999996},"last":0.45700000000000002},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.497},"min":{"1min":0.53600000000000003,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.68600000000000005,"5min":0.75600000000000001,"15min":0.82899999999999996},"last":0.58099999999999996}]}]},{"osd":0,"up_from":0,"seq":201863463006,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084397,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284748,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":69311,"kb_avail":94087092,"statfs":{"total":96636764160,"available":96345182208,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":70974516},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:08:07 2021","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44400000000000001,"15min":0.45600000000000002},"min":{"1min":0.36499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.49099999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.52000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66300000000000003,"5min":0.69099999999999995,"15min":0.80400000000000005},"last":0.627}]},{"osd":2,"last update":"Fri Feb 12 18:07:39 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.436},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.60099999999999998,"5min":0.63800000000000001,"15min":0.78900000000000003},"last":0.436},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.755},"last":0.48799999999999999}]}]}]}} 2021-02-12T18:08:15.043 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:15.044 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:15.044 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:15.044 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:15.044 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:15.044 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:15.044 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:15.044 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:15.044 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:08:35.045 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:08:35.336 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:08:35.337 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:08:35.347 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":381,"stamp":"2021-02-12T18:08:34.493299+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55536,"ondisk_log_size":55536,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":585560,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":154687,"kb_avail":188158120,"statfs":{"total":193273528320,"available":192673914880,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":158399540},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001401"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8507","reported_seq":"14071","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:08:29.683528+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:08:29.683528+0000","last_peered":"2021-02-12T18:08:29.683528+0000","last_clean":"2021-02-12T18:08:29.683528+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:08:29.683528+0000","last_undegraded":"2021-02-12T18:08:29.683528+0000","last_fullsized":"2021-02-12T18:08:29.683528+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8507,"ondisk_log_size":8507,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55536,"ondisk_log_size":55536,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738508,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300748,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":85312,"kb_avail":94071092,"statfs":{"total":96636764160,"available":96328798208,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":87359488},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:08:00 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.43099999999999999,"15min":0.45500000000000002},"min":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.51500000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.39900000000000002},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.35999999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.442}]},{"osd":1,"last update":"Fri Feb 12 18:07:41 2021","interfaces":[{"interface":"back","average":{"1min":0.46999999999999997,"5min":0.46999999999999997,"15min":0.42599999999999999},"min":{"1min":0.377,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.65500000000000003,"5min":0.65500000000000003,"15min":0.70399999999999996},"last":0.36499999999999999},{"interface":"front","average":{"1min":0.60399999999999998,"5min":0.60499999999999998,"15min":0.497},"min":{"1min":0.53600000000000003,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.68600000000000005,"5min":0.75600000000000001,"15min":0.82899999999999996},"last":0.51600000000000001}]}]},{"osd":0,"up_from":0,"seq":201863463010,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084401,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":284812,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":69375,"kb_avail":94087028,"statfs":{"total":96636764160,"available":96345116672,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":71040052},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:08:07 2021","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44400000000000001,"15min":0.45600000000000002},"min":{"1min":0.36499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.49099999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.443},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.52000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66300000000000003,"5min":0.69099999999999995,"15min":0.80400000000000005},"last":0.60099999999999998}]},{"osd":2,"last update":"Fri Feb 12 18:07:39 2021","interfaces":[{"interface":"back","average":{"1min":0.48099999999999998,"5min":0.48499999999999999,"15min":0.436},"min":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.60099999999999998,"5min":0.63800000000000001,"15min":0.78900000000000003},"last":0.47199999999999998},{"interface":"front","average":{"1min":0.499,"5min":0.501,"15min":0.51600000000000001},"min":{"1min":0.40300000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.66000000000000003,"5min":0.66000000000000003,"15min":0.755},"last":0.52100000000000002}]}]}]}} 2021-02-12T18:08:35.348 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:35.349 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:35.349 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:35.349 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:35.349 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:35.349 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:35.349 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:35.349 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:35.349 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:08:55.350 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:08:55.650 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:08:55.652 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:08:55.662 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":391,"stamp":"2021-02-12T18:08:54.495635+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55611,"ondisk_log_size":55611,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":589784,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":158911,"kb_avail":188153896,"statfs":{"total":193273528320,"available":192669589504,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":162724916},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001454"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8582","reported_seq":"14146","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:08:48.942816+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:08:48.942816+0000","last_peered":"2021-02-12T18:08:48.942816+0000","last_clean":"2021-02-12T18:08:48.942816+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:08:48.942816+0000","last_undegraded":"2021-02-12T18:08:48.942816+0000","last_fullsized":"2021-02-12T18:08:48.942816+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8582,"ondisk_log_size":8582,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55611,"ondisk_log_size":55611,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738512,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300812,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":85376,"kb_avail":94071028,"statfs":{"total":96636764160,"available":96328732672,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":87425024},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:08:00 2021","interfaces":[{"interface":"back","average":{"1min":0.434,"5min":0.43099999999999999,"15min":0.45500000000000002},"min":{"1min":0.34899999999999998,"5min":0.33900000000000002,"15min":0.33900000000000002},"max":{"1min":0.51500000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.371},{"interface":"front","average":{"1min":0.46800000000000003,"5min":0.46400000000000002,"15min":0.50900000000000001},"min":{"1min":0.35999999999999999,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.67500000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40899999999999997}]},{"osd":1,"last update":"Fri Feb 12 18:08:49 2021","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46600000000000003,"15min":0.432},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.60699999999999998,"5min":0.65500000000000003,"15min":0.70399999999999996},"last":0.44},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60399999999999998,"15min":0.51000000000000001},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.65000000000000002}]}]},{"osd":0,"up_from":0,"seq":201863463014,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084405,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":288972,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":73535,"kb_avail":94082868,"statfs":{"total":96636764160,"available":96340856832,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":75299892},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:08:07 2021","interfaces":[{"interface":"back","average":{"1min":0.42999999999999999,"5min":0.44400000000000001,"15min":0.45600000000000002},"min":{"1min":0.36499999999999999,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.49099999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.43099999999999999},{"interface":"front","average":{"1min":0.58399999999999996,"5min":0.54700000000000004,"15min":0.52900000000000003},"min":{"1min":0.52000000000000002,"5min":0.36499999999999999,"15min":0.36499999999999999},"max":{"1min":0.66300000000000003,"5min":0.69099999999999995,"15min":0.80400000000000005},"last":0.52800000000000002}]},{"osd":2,"last update":"Fri Feb 12 18:08:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.442},"min":{"1min":0.38,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.63800000000000001,"15min":0.78900000000000003},"last":0.503},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.755},"last":0.58299999999999996}]}]}]}} 2021-02-12T18:08:55.663 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:55.663 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:55.664 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:55.664 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:55.664 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:55.664 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:55.664 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:55.664 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:08:55.664 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.7 2021-02-12T18:08:55.965 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.7 on osd.1 to deep-scrub 2021-02-12T18:08:55.975 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.6 2021-02-12T18:08:56.295 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.6 on osd.1 to deep-scrub 2021-02-12T18:08:56.306 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.1 2021-02-12T18:08:56.612 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.1 on osd.2 to deep-scrub 2021-02-12T18:08:56.622 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2021-02-12T18:08:56.926 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2021-02-12T18:08:56.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.2 2021-02-12T18:08:57.237 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.2 on osd.1 to deep-scrub 2021-02-12T18:08:57.247 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.3 2021-02-12T18:08:57.548 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.3 on osd.1 to deep-scrub 2021-02-12T18:08:57.562 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.4 2021-02-12T18:08:57.863 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.4 on osd.1 to deep-scrub 2021-02-12T18:08:57.873 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.5 2021-02-12T18:08:58.174 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.5 on osd.2 to deep-scrub 2021-02-12T18:08:58.185 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:09:18.186 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:09:18.478 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:09:18.480 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:09:18.490 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":402,"stamp":"2021-02-12T18:09:16.498559+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55686,"ondisk_log_size":55686,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":589912,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":159039,"kb_avail":188153768,"statfs":{"total":193273528320,"available":192669458432,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":162855988},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001610"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8657","reported_seq":"14221","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:09:08.202166+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:09:08.202166+0000","last_peered":"2021-02-12T18:09:08.202166+0000","last_clean":"2021-02-12T18:09:08.202166+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:09:08.202166+0000","last_undegraded":"2021-02-12T18:09:08.202166+0000","last_fullsized":"2021-02-12T18:09:08.202166+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8657,"ondisk_log_size":8657,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55686,"ondisk_log_size":55686,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738516,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":300876,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":85440,"kb_avail":94070964,"statfs":{"total":96636764160,"available":96328667136,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":87490560},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:09:02 2021","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42599999999999999,"15min":0.45100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.51000000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.504},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.54500000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.48499999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:08:49 2021","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46600000000000003,"15min":0.432},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.60699999999999998,"5min":0.65500000000000003,"15min":0.70399999999999996},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60399999999999998,"15min":0.51000000000000001},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.626}]}]},{"osd":0,"up_from":0,"seq":201863463018,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084409,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289036,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":73599,"kb_avail":94082804,"statfs":{"total":96636764160,"available":96340791296,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":75365428},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:09:10 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.439,"15min":0.45400000000000001},"min":{"1min":0.39500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.47999999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40300000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.53500000000000003},"min":{"1min":0.52800000000000002,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.58199999999999996}]},{"osd":2,"last update":"Fri Feb 12 18:08:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.442},"min":{"1min":0.38,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.63800000000000001,"15min":0.78900000000000003},"last":0.49099999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.755},"last":0.433}]}]}]}} 2021-02-12T18:09:18.491 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:18.491 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:18.491 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:18.492 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:18.492 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:18.492 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:18.492 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:18.492 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:18.492 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:09:38.493 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:09:38.798 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:09:38.799 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:09:38.810 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":413,"stamp":"2021-02-12T18:09:38.501369+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55786,"ondisk_log_size":55786,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594264,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163391,"kb_avail":188149416,"statfs":{"total":193273528320,"available":192665001984,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167312436},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001528"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8757","reported_seq":"14321","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:09:33.880067+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:09:33.880067+0000","last_peered":"2021-02-12T18:09:33.880067+0000","last_clean":"2021-02-12T18:09:33.880067+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:09:33.880067+0000","last_undegraded":"2021-02-12T18:09:33.880067+0000","last_fullsized":"2021-02-12T18:09:33.880067+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8757,"ondisk_log_size":8757,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55786,"ondisk_log_size":55786,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738520,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305100,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":89664,"kb_avail":94066740,"statfs":{"total":96636764160,"available":96324341760,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91815936},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:09:02 2021","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42599999999999999,"15min":0.45100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.51000000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.504},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.54500000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.44400000000000001}]},{"osd":1,"last update":"Fri Feb 12 18:08:49 2021","interfaces":[{"interface":"back","average":{"1min":0.45700000000000002,"5min":0.46600000000000003,"15min":0.432},"min":{"1min":0.36499999999999999,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.60699999999999998,"5min":0.65500000000000003,"15min":0.70399999999999996},"last":0.46200000000000002},{"interface":"front","average":{"1min":0.59899999999999998,"5min":0.60399999999999998,"15min":0.51000000000000001},"min":{"1min":0.51600000000000001,"5min":0.39000000000000001,"15min":0.311},"max":{"1min":0.84899999999999998,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58599999999999997}]}]},{"osd":0,"up_from":0,"seq":201863463023,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084414,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289164,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":73727,"kb_avail":94082676,"statfs":{"total":96636764160,"available":96340660224,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":75496500},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:09:10 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.439,"15min":0.45400000000000001},"min":{"1min":0.39500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.47999999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.53500000000000003},"min":{"1min":0.52800000000000002,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.64600000000000002}]},{"osd":2,"last update":"Fri Feb 12 18:08:39 2021","interfaces":[{"interface":"back","average":{"1min":0.47499999999999998,"5min":0.48599999999999999,"15min":0.442},"min":{"1min":0.38,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.623,"5min":0.63800000000000001,"15min":0.78900000000000003},"last":0.52200000000000002},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.5,"15min":0.51600000000000001},"min":{"1min":0.39200000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.755},"last":0.433}]}]}]}} 2021-02-12T18:09:38.811 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:38.811 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:38.811 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:38.811 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:38.811 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:38.811 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:38.811 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:38.811 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:38.812 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:09:58.813 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:09:59.108 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:09:59.110 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:09:59.121 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":423,"stamp":"2021-02-12T18:09:58.504024+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55861,"ondisk_log_size":55861,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594392,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163519,"kb_avail":188149288,"statfs":{"total":193273528320,"available":192664870912,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167443508},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001539"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8832","reported_seq":"14396","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:09:53.145996+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:09:53.145996+0000","last_peered":"2021-02-12T18:09:53.145996+0000","last_clean":"2021-02-12T18:09:53.145996+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:09:53.145996+0000","last_undegraded":"2021-02-12T18:09:53.145996+0000","last_fullsized":"2021-02-12T18:09:53.145996+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8832,"ondisk_log_size":8832,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55861,"ondisk_log_size":55861,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738524,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305164,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":89728,"kb_avail":94066676,"statfs":{"total":96636764160,"available":96324276224,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91881472},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:09:02 2021","interfaces":[{"interface":"back","average":{"1min":0.41199999999999998,"5min":0.42599999999999999,"15min":0.45100000000000001},"min":{"1min":0.33700000000000002,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.51000000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.496},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.46400000000000002,"15min":0.504},"min":{"1min":0.40899999999999997,"5min":0.35799999999999998,"15min":0.35799999999999998},"max":{"1min":0.54500000000000004,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.46999999999999997}]},{"osd":1,"last update":"Fri Feb 12 18:09:50 2021","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.438},"min":{"1min":0.39100000000000001,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.443},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60299999999999998,"15min":0.52400000000000002},"min":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.60399999999999998}]}]},{"osd":0,"up_from":0,"seq":201863463027,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084418,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289228,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":73791,"kb_avail":94082612,"statfs":{"total":96636764160,"available":96340594688,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":75562036},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:09:10 2021","interfaces":[{"interface":"back","average":{"1min":0.437,"5min":0.439,"15min":0.45400000000000001},"min":{"1min":0.39500000000000002,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.47999999999999998,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.60899999999999999,"5min":0.56499999999999995,"15min":0.53500000000000003},"min":{"1min":0.52800000000000002,"5min":0.39200000000000002,"15min":0.36499999999999999},"max":{"1min":0.78900000000000003,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.56299999999999994}]},{"osd":2,"last update":"Fri Feb 12 18:09:40 2021","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48899999999999999,"15min":0.44900000000000001},"min":{"1min":0.41099999999999998,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.78900000000000003},"last":0.438},{"interface":"front","average":{"1min":0.5,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.755},"last":0.499}]}]}]}} 2021-02-12T18:09:59.121 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:59.122 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:59.122 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:59.122 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:59.122 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:59.122 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:59.122 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:59.122 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:09:59.122 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:10:19.124 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:10:19.418 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:10:19.420 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:10:19.430 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":433,"stamp":"2021-02-12T18:10:18.507027+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55936,"ondisk_log_size":55936,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":594520,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":163647,"kb_avail":188149160,"statfs":{"total":193273528320,"available":192664739840,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":167574580},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002036"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8907","reported_seq":"14471","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:10:12.398550+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:10:12.398550+0000","last_peered":"2021-02-12T18:10:12.398550+0000","last_clean":"2021-02-12T18:10:12.398550+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:10:12.398550+0000","last_undegraded":"2021-02-12T18:10:12.398550+0000","last_fullsized":"2021-02-12T18:10:12.398550+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8907,"ondisk_log_size":8907,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":55936,"ondisk_log_size":55936,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738528,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305228,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":89792,"kb_avail":94066612,"statfs":{"total":96636764160,"available":96324210688,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":91947008},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:10:05 2021","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42399999999999999,"15min":0.44700000000000001},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.51000000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46100000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35799999999999998},"max":{"1min":0.51800000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.44500000000000001}]},{"osd":1,"last update":"Fri Feb 12 18:09:50 2021","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.438},"min":{"1min":0.39100000000000001,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.46500000000000002},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60299999999999998,"15min":0.52400000000000002},"min":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.61299999999999999}]}]},{"osd":0,"up_from":0,"seq":201863463031,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084422,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289292,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":73855,"kb_avail":94082548,"statfs":{"total":96636764160,"available":96340529152,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":75627572},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:10:10 2021","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.435,"15min":0.45100000000000001},"min":{"1min":0.35399999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.47099999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.50700000000000001,"5min":0.39800000000000002,"15min":0.36499999999999999},"max":{"1min":0.64600000000000002,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Feb 12 18:09:40 2021","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48899999999999999,"15min":0.44900000000000001},"min":{"1min":0.41099999999999998,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.78900000000000003},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.5,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.755},"last":0.42899999999999999}]}]}]}} 2021-02-12T18:10:19.431 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:19.431 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:19.431 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:19.431 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:19.432 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:19.432 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:19.432 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:19.432 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:19.432 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:10:39.433 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:10:39.731 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:10:39.733 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:10:39.744 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":443,"stamp":"2021-02-12T18:10:38.509684+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56011,"ondisk_log_size":56011,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":598744,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":167871,"kb_avail":188144936,"statfs":{"total":193273528320,"available":192660414464,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":171899956},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001611"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'8982","reported_seq":"14546","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:10:31.657187+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:10:31.657187+0000","last_peered":"2021-02-12T18:10:31.657187+0000","last_clean":"2021-02-12T18:10:31.657187+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:10:31.657187+0000","last_undegraded":"2021-02-12T18:10:31.657187+0000","last_fullsized":"2021-02-12T18:10:31.657187+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":8982,"ondisk_log_size":8982,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56011,"ondisk_log_size":56011,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738532,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309388,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":93952,"kb_avail":94062452,"statfs":{"total":96636764160,"available":96319950848,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96206848},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:10:05 2021","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42399999999999999,"15min":0.44700000000000001},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.51000000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46100000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35799999999999998},"max":{"1min":0.51800000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.40100000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:09:50 2021","interfaces":[{"interface":"back","average":{"1min":0.47599999999999998,"5min":0.46600000000000003,"15min":0.438},"min":{"1min":0.39100000000000001,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.60599999999999998,"5min":0.60299999999999998,"15min":0.52400000000000002},"min":{"1min":0.51400000000000001,"5min":0.51200000000000001,"15min":0.311},"max":{"1min":0.69799999999999995,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53100000000000003}]}]},{"osd":0,"up_from":0,"seq":201863463035,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084426,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":289356,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":73919,"kb_avail":94082484,"statfs":{"total":96636764160,"available":96340463616,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":75693108},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:10:10 2021","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.435,"15min":0.45100000000000001},"min":{"1min":0.35399999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.47099999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42699999999999999},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.50700000000000001,"5min":0.39800000000000002,"15min":0.36499999999999999},"max":{"1min":0.64600000000000002,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.56499999999999995}]},{"osd":2,"last update":"Fri Feb 12 18:09:40 2021","interfaces":[{"interface":"back","average":{"1min":0.502,"5min":0.48899999999999999,"15min":0.44900000000000001},"min":{"1min":0.41099999999999998,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.67400000000000004,"5min":0.67400000000000004,"15min":0.78900000000000003},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.5,"5min":0.498,"15min":0.51600000000000001},"min":{"1min":0.40500000000000003,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.71299999999999997,"5min":0.71299999999999997,"15min":0.755},"last":0.45900000000000002}]}]}]}} 2021-02-12T18:10:39.745 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:39.745 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:39.745 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:39.745 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:39.745 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:39.745 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:39.745 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:39.745 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:10:39.745 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:10:59.746 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:11:00.047 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:11:00.049 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:11:00.059 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":453,"stamp":"2021-02-12T18:10:58.512352+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56111,"ondisk_log_size":56111,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603096,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":172223,"kb_avail":188140584,"statfs":{"total":193273528320,"available":192655958016,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":176356404},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001572"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9082","reported_seq":"14646","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:10:57.329576+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:10:57.329576+0000","last_peered":"2021-02-12T18:10:57.329576+0000","last_clean":"2021-02-12T18:10:57.329576+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:10:57.329576+0000","last_undegraded":"2021-02-12T18:10:57.329576+0000","last_fullsized":"2021-02-12T18:10:57.329576+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9082,"ondisk_log_size":9082,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56111,"ondisk_log_size":56111,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738536,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309516,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":94080,"kb_avail":94062324,"statfs":{"total":96636764160,"available":96319819776,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96337920},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:10:05 2021","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42399999999999999,"15min":0.44700000000000001},"min":{"1min":0.34300000000000003,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.51000000000000001,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.39500000000000002},{"interface":"front","average":{"1min":0.44600000000000001,"5min":0.46100000000000002,"15min":0.498},"min":{"1min":0.38200000000000001,"5min":0.35999999999999999,"15min":0.35799999999999998},"max":{"1min":0.51800000000000002,"5min":0.79300000000000004,"15min":0.79300000000000004},"last":0.433}]},{"osd":1,"last update":"Fri Feb 12 18:10:51 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46899999999999997,"15min":0.44400000000000001},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.63700000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.434},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.53700000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.311},"max":{"1min":0.746,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58999999999999997}]}]},{"osd":0,"up_from":0,"seq":201863463039,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084430,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293580,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":78143,"kb_avail":94078260,"statfs":{"total":96636764160,"available":96336138240,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":80018484},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:10:10 2021","interfaces":[{"interface":"back","average":{"1min":0.41799999999999998,"5min":0.435,"15min":0.45100000000000001},"min":{"1min":0.35399999999999998,"5min":0.32600000000000001,"15min":0.32600000000000001},"max":{"1min":0.47099999999999997,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.56899999999999995,"5min":0.57399999999999995,"15min":0.53900000000000003},"min":{"1min":0.50700000000000001,"5min":0.39800000000000002,"15min":0.36499999999999999},"max":{"1min":0.64600000000000002,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.61499999999999999}]},{"osd":2,"last update":"Fri Feb 12 18:10:44 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.45500000000000002},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.57699999999999996,"5min":0.67400000000000004,"15min":0.78900000000000003},"last":0.48599999999999999},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.496,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.55600000000000005,"5min":0.71299999999999997,"15min":0.755},"last":0.40300000000000002}]}]}]}} 2021-02-12T18:11:00.060 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:00.060 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:00.060 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:00.060 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:00.060 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:00.060 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:00.060 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:00.061 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:00.061 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.7 2021-02-12T18:11:00.359 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.7 on osd.1 to deep-scrub 2021-02-12T18:11:00.369 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.6 2021-02-12T18:11:00.668 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.6 on osd.1 to deep-scrub 2021-02-12T18:11:00.678 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.1 2021-02-12T18:11:00.979 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.1 on osd.2 to deep-scrub 2021-02-12T18:11:00.989 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2021-02-12T18:11:01.286 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2021-02-12T18:11:01.296 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.2 2021-02-12T18:11:01.608 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.2 on osd.1 to deep-scrub 2021-02-12T18:11:01.618 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.3 2021-02-12T18:11:01.920 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.3 on osd.1 to deep-scrub 2021-02-12T18:11:01.930 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.4 2021-02-12T18:11:02.225 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.4 on osd.1 to deep-scrub 2021-02-12T18:11:02.235 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.5 2021-02-12T18:11:02.531 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.5 on osd.2 to deep-scrub 2021-02-12T18:11:02.541 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:11:22.542 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:11:22.852 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:11:22.854 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:11:22.864 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":465,"stamp":"2021-02-12T18:11:22.515580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56186,"ondisk_log_size":56186,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":603224,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":172351,"kb_avail":188140456,"statfs":{"total":193273528320,"available":192655826944,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":176487476},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001608"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9157","reported_seq":"14721","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:11:16.594772+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:11:16.594772+0000","last_peered":"2021-02-12T18:11:16.594772+0000","last_clean":"2021-02-12T18:11:16.594772+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:11:16.594772+0000","last_undegraded":"2021-02-12T18:11:16.594772+0000","last_fullsized":"2021-02-12T18:11:16.594772+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9157,"ondisk_log_size":9157,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56186,"ondisk_log_size":56186,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738541,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":309580,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":94144,"kb_avail":94062260,"statfs":{"total":96636764160,"available":96319754240,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":96403456},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:11:07 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.42499999999999999,"15min":0.44500000000000001},"min":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.60499999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.373},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.49199999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.59099999999999997,"5min":0.67500000000000004,"15min":0.79300000000000004},"last":0.41299999999999998}]},{"osd":1,"last update":"Fri Feb 12 18:10:51 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46899999999999997,"15min":0.44400000000000001},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.63700000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45300000000000001},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.53700000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.311},"max":{"1min":0.746,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.56599999999999995}]}]},{"osd":0,"up_from":0,"seq":201863463044,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084434,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293644,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":78207,"kb_avail":94078196,"statfs":{"total":96636764160,"available":96336072704,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":80084020},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[0,0,0,0,2],"upper_bound":32},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:11:11 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.432,"15min":0.44800000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.49299999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.54300000000000004},"min":{"1min":0.52300000000000002,"5min":0.43099999999999999,"15min":0.36499999999999999},"max":{"1min":0.64000000000000001,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.53900000000000003}]},{"osd":2,"last update":"Fri Feb 12 18:10:44 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.45500000000000002},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.57699999999999996,"5min":0.67400000000000004,"15min":0.78900000000000003},"last":0.41199999999999998},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.496,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.55600000000000005,"5min":0.71299999999999997,"15min":0.755},"last":0.49099999999999999}]}]}]}} 2021-02-12T18:11:22.865 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:22.866 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:22.866 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:22.866 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:22.866 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:22.866 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:22.866 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:22.866 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:22.866 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:11:42.868 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:11:43.162 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:11:43.164 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:11:43.174 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":475,"stamp":"2021-02-12T18:11:42.518134+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56261,"ondisk_log_size":56261,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":607448,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":176575,"kb_avail":188136232,"statfs":{"total":193273528320,"available":192651501568,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":180812852},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001490"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9232","reported_seq":"14796","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:11:35.853948+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:11:35.853948+0000","last_peered":"2021-02-12T18:11:35.853948+0000","last_clean":"2021-02-12T18:11:35.853948+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:11:35.853948+0000","last_undegraded":"2021-02-12T18:11:35.853948+0000","last_fullsized":"2021-02-12T18:11:35.853948+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9232,"ondisk_log_size":9232,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56261,"ondisk_log_size":56261,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738545,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":313740,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":98304,"kb_avail":94058100,"statfs":{"total":96636764160,"available":96315494400,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":100663296},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:11:07 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.42499999999999999,"15min":0.44500000000000001},"min":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.60499999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.49199999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.59099999999999997,"5min":0.67500000000000004,"15min":0.79300000000000004},"last":0.47299999999999998}]},{"osd":1,"last update":"Fri Feb 12 18:10:51 2021","interfaces":[{"interface":"back","average":{"1min":0.46500000000000002,"5min":0.46899999999999997,"15min":0.44400000000000001},"min":{"1min":0.371,"5min":0.35499999999999998,"15min":0.28299999999999997},"max":{"1min":0.63700000000000001,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.60499999999999998,"5min":0.60499999999999998,"15min":0.53700000000000003},"min":{"1min":0.50700000000000001,"5min":0.50700000000000001,"15min":0.311},"max":{"1min":0.746,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.55500000000000005}]}]},{"osd":0,"up_from":0,"seq":201863463048,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084439,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293708,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":78271,"kb_avail":94078132,"statfs":{"total":96636764160,"available":96336007168,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":80149556},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:11:11 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.432,"15min":0.44800000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.49299999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.42499999999999999},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.54300000000000004},"min":{"1min":0.52300000000000002,"5min":0.43099999999999999,"15min":0.36499999999999999},"max":{"1min":0.64000000000000001,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.56299999999999994}]},{"osd":2,"last update":"Fri Feb 12 18:10:44 2021","interfaces":[{"interface":"back","average":{"1min":0.48599999999999999,"5min":0.48999999999999999,"15min":0.45500000000000002},"min":{"1min":0.38700000000000001,"5min":0.35299999999999998,"15min":0.27300000000000002},"max":{"1min":0.57699999999999996,"5min":0.67400000000000004,"15min":0.78900000000000003},"last":0.46700000000000003},{"interface":"front","average":{"1min":0.46500000000000002,"5min":0.496,"15min":0.51300000000000001},"min":{"1min":0.40100000000000002,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":0.55600000000000005,"5min":0.71299999999999997,"15min":0.755},"last":0.36599999999999999}]}]}]}} 2021-02-12T18:11:43.175 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:43.175 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:43.175 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:43.176 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:43.176 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:43.176 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:43.176 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:43.176 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:11:43.176 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:12:03.177 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:12:03.489 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:12:03.491 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:12:03.501 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":485,"stamp":"2021-02-12T18:12:02.520718+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56336,"ondisk_log_size":56336,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":591192,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":160319,"kb_avail":188152488,"statfs":{"total":193273528320,"available":192668147712,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":164166708},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001563"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9307","reported_seq":"14871","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:11:55.108613+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:11:55.108613+0000","last_peered":"2021-02-12T18:11:55.108613+0000","last_clean":"2021-02-12T18:11:55.108613+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:11:55.108613+0000","last_undegraded":"2021-02-12T18:11:55.108613+0000","last_fullsized":"2021-02-12T18:11:55.108613+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9307,"ondisk_log_size":9307,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56336,"ondisk_log_size":56336,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738549,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297420,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":81984,"kb_avail":94074420,"statfs":{"total":96636764160,"available":96332206080,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":83951616},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:11:07 2021","interfaces":[{"interface":"back","average":{"1min":0.44500000000000001,"5min":0.42499999999999999,"15min":0.44500000000000001},"min":{"1min":0.36699999999999999,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.60499999999999998,"5min":0.67200000000000004,"15min":0.67200000000000004},"last":0.45400000000000001},{"interface":"front","average":{"1min":0.46100000000000002,"5min":0.46000000000000002,"15min":0.49199999999999999},"min":{"1min":0.35699999999999998,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.59099999999999997,"5min":0.67500000000000004,"15min":0.79300000000000004},"last":0.5}]},{"osd":1,"last update":"Fri Feb 12 18:11:55 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46300000000000002,"15min":0.44800000000000001},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.28299999999999997},"max":{"1min":0.52000000000000002,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47899999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59899999999999998,"15min":0.54900000000000004},"min":{"1min":0.54800000000000004,"5min":0.50700000000000001,"15min":0.311},"max":{"1min":0.71299999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58899999999999997}]}]},{"osd":0,"up_from":0,"seq":201863463052,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084443,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293772,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":78335,"kb_avail":94078068,"statfs":{"total":96636764160,"available":96335941632,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":80215092},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:11:11 2021","interfaces":[{"interface":"back","average":{"1min":0.42499999999999999,"5min":0.432,"15min":0.44800000000000001},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.49299999999999999,"5min":0.70199999999999996,"15min":0.70199999999999996},"last":0.40999999999999998},{"interface":"front","average":{"1min":0.58099999999999996,"5min":0.58199999999999996,"15min":0.54300000000000004},"min":{"1min":0.52300000000000002,"5min":0.43099999999999999,"15min":0.36499999999999999},"max":{"1min":0.64000000000000001,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.59099999999999997}]},{"osd":2,"last update":"Fri Feb 12 18:11:47 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48399999999999999,"15min":0.46100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.27300000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.56000000000000005},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48899999999999999,"15min":0.51000000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.56999999999999995,"5min":0.71299999999999997,"15min":0.755},"last":0.53100000000000003}]}]}]}} 2021-02-12T18:12:03.502 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:03.502 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:03.502 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:03.503 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:03.503 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:03.503 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:03.503 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:03.503 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:03.503 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:12:23.505 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:12:23.813 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:12:23.815 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:12:23.825 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":495,"stamp":"2021-02-12T18:12:22.523343+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56411,"ondisk_log_size":56411,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":591384,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":160511,"kb_avail":188152296,"statfs":{"total":193273528320,"available":192667951104,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":164363316},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001533"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9382","reported_seq":"14946","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:12:14.383465+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:12:14.383465+0000","last_peered":"2021-02-12T18:12:14.383465+0000","last_clean":"2021-02-12T18:12:14.383465+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:12:14.383465+0000","last_undegraded":"2021-02-12T18:12:14.383465+0000","last_fullsized":"2021-02-12T18:12:14.383465+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9382,"ondisk_log_size":9382,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56411,"ondisk_log_size":56411,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738553,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":297548,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":82112,"kb_avail":94074292,"statfs":{"total":96636764160,"available":96332075008,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":84082688},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:12:09 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42999999999999999,"15min":0.44400000000000001},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.54700000000000004,"5min":0.60499999999999998,"15min":0.67200000000000004},"last":0.49299999999999999},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.46500000000000002,"15min":0.48899999999999999},"min":{"1min":0.39500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.59299999999999997,"5min":0.67500000000000004,"15min":0.79300000000000004},"last":0.45800000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:11:55 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46300000000000002,"15min":0.44800000000000001},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.28299999999999997},"max":{"1min":0.52000000000000002,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.47699999999999998},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59899999999999998,"15min":0.54900000000000004},"min":{"1min":0.54800000000000004,"5min":0.50700000000000001,"15min":0.311},"max":{"1min":0.71299999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53700000000000003}]}]},{"osd":0,"up_from":0,"seq":201863463056,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084447,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293836,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":78399,"kb_avail":94078004,"statfs":{"total":96636764160,"available":96335876096,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":80280628},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:12:15 2021","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42299999999999999,"15min":0.44500000000000001},"min":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.502,"5min":0.502,"15min":0.70199999999999996},"last":0.36699999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57899999999999996,"15min":0.54500000000000004},"min":{"1min":0.52100000000000002,"5min":0.50700000000000001,"15min":0.36499999999999999},"max":{"1min":0.626,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.56000000000000005}]},{"osd":2,"last update":"Fri Feb 12 18:11:47 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48399999999999999,"15min":0.46100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.27300000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.44700000000000001},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48899999999999999,"15min":0.51000000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.56999999999999995,"5min":0.71299999999999997,"15min":0.755},"last":0.60499999999999998}]}]}]}} 2021-02-12T18:12:23.826 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:23.826 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:23.826 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:23.827 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:23.827 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:23.827 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:23.827 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:23.827 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:23.827 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:12:43.828 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:12:44.122 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:12:44.123 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:12:44.134 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":505,"stamp":"2021-02-12T18:12:42.525915+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56486,"ondisk_log_size":56486,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595608,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":164735,"kb_avail":188148072,"statfs":{"total":193273528320,"available":192663625728,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":168688692},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001493"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9457","reported_seq":"15021","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:12:33.633030+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:12:33.633030+0000","last_peered":"2021-02-12T18:12:33.633030+0000","last_clean":"2021-02-12T18:12:33.633030+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:12:33.633030+0000","last_undegraded":"2021-02-12T18:12:33.633030+0000","last_fullsized":"2021-02-12T18:12:33.633030+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9457,"ondisk_log_size":9457,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56486,"ondisk_log_size":56486,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738557,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301708,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":86272,"kb_avail":94070132,"statfs":{"total":96636764160,"available":96327815168,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88342528},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:12:09 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42999999999999999,"15min":0.44400000000000001},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.54700000000000004,"5min":0.60499999999999998,"15min":0.67200000000000004},"last":0.39600000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.46500000000000002,"15min":0.48899999999999999},"min":{"1min":0.39500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.59299999999999997,"5min":0.67500000000000004,"15min":0.79300000000000004},"last":0.46000000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:11:55 2021","interfaces":[{"interface":"back","average":{"1min":0.44900000000000001,"5min":0.46300000000000002,"15min":0.44800000000000001},"min":{"1min":0.378,"5min":0.36499999999999999,"15min":0.28299999999999997},"max":{"1min":0.52000000000000002,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.42799999999999999},{"interface":"front","average":{"1min":0.58499999999999996,"5min":0.59899999999999998,"15min":0.54900000000000004},"min":{"1min":0.54800000000000004,"5min":0.50700000000000001,"15min":0.311},"max":{"1min":0.71299999999999997,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.58899999999999997}]}]},{"osd":0,"up_from":0,"seq":201863463060,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084451,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293900,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":78463,"kb_avail":94077940,"statfs":{"total":96636764160,"available":96335810560,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":80346164},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:12:15 2021","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42299999999999999,"15min":0.44500000000000001},"min":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.502,"5min":0.502,"15min":0.70199999999999996},"last":0.42299999999999999},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57899999999999996,"15min":0.54500000000000004},"min":{"1min":0.52100000000000002,"5min":0.50700000000000001,"15min":0.36499999999999999},"max":{"1min":0.626,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.57799999999999996}]},{"osd":2,"last update":"Fri Feb 12 18:11:47 2021","interfaces":[{"interface":"back","average":{"1min":0.47799999999999998,"5min":0.48399999999999999,"15min":0.46100000000000002},"min":{"1min":0.34200000000000003,"5min":0.34200000000000003,"15min":0.27300000000000002},"max":{"1min":0.91600000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.53300000000000003},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.48899999999999999,"15min":0.51000000000000001},"min":{"1min":0.36599999999999999,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.56999999999999995,"5min":0.71299999999999997,"15min":0.755},"last":0.39300000000000002}]}]}]}} 2021-02-12T18:12:44.135 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:44.135 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:44.135 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:44.135 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:44.135 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:44.135 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:44.135 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:44.135 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:12:44.136 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:13:04.137 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:13:04.435 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:13:04.436 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:13:04.447 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":515,"stamp":"2021-02-12T18:13:02.528504+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56561,"ondisk_log_size":56561,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":595736,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":164863,"kb_avail":188147944,"statfs":{"total":193273528320,"available":192663494656,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":168819764},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001554"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9532","reported_seq":"15096","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:12:52.887034+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:12:52.887034+0000","last_peered":"2021-02-12T18:12:52.887034+0000","last_clean":"2021-02-12T18:12:52.887034+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:12:52.887034+0000","last_undegraded":"2021-02-12T18:12:52.887034+0000","last_fullsized":"2021-02-12T18:12:52.887034+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9532,"ondisk_log_size":9532,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56561,"ondisk_log_size":56561,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738562,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301772,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":86336,"kb_avail":94070068,"statfs":{"total":96636764160,"available":96327749632,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88408064},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:12:09 2021","interfaces":[{"interface":"back","average":{"1min":0.45300000000000001,"5min":0.42999999999999999,"15min":0.44400000000000001},"min":{"1min":0.373,"5min":0.33700000000000002,"15min":0.33700000000000002},"max":{"1min":0.54700000000000004,"5min":0.60499999999999998,"15min":0.67200000000000004},"last":0.33300000000000002},{"interface":"front","average":{"1min":0.48199999999999998,"5min":0.46500000000000002,"15min":0.48899999999999999},"min":{"1min":0.39500000000000002,"5min":0.35699999999999998,"15min":0.35699999999999998},"max":{"1min":0.59299999999999997,"5min":0.67500000000000004,"15min":0.79300000000000004},"last":0.41299999999999998}]},{"osd":1,"last update":"Fri Feb 12 18:12:57 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46200000000000002,"15min":0.45400000000000001},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.28299999999999997},"max":{"1min":0.56200000000000006,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45100000000000001},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.311},"max":{"1min":0.66700000000000004,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.48899999999999999}]}]},{"osd":0,"up_from":0,"seq":201863463064,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084455,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":293964,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":78527,"kb_avail":94077876,"statfs":{"total":96636764160,"available":96335745024,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":80411700},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:12:15 2021","interfaces":[{"interface":"back","average":{"1min":0.40799999999999997,"5min":0.42299999999999999,"15min":0.44500000000000001},"min":{"1min":0.36299999999999999,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.502,"5min":0.502,"15min":0.70199999999999996},"last":0.56799999999999995},{"interface":"front","average":{"1min":0.55600000000000005,"5min":0.57899999999999996,"15min":0.54500000000000004},"min":{"1min":0.52100000000000002,"5min":0.50700000000000001,"15min":0.36499999999999999},"max":{"1min":0.626,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.59599999999999997}]},{"osd":2,"last update":"Fri Feb 12 18:12:52 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.27300000000000002},"max":{"1min":0.63300000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46000000000000002},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48299999999999998,"15min":0.50800000000000001},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.755},"last":0.502}]}]}]}} 2021-02-12T18:13:04.448 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:04.448 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:04.448 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:04.448 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:04.448 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:04.448 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:04.448 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:04.448 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:04.449 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.7 2021-02-12T18:13:04.749 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.7 on osd.1 to deep-scrub 2021-02-12T18:13:04.760 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.6 2021-02-12T18:13:05.055 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.6 on osd.1 to deep-scrub 2021-02-12T18:13:05.065 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.1 2021-02-12T18:13:05.367 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.1 on osd.2 to deep-scrub 2021-02-12T18:13:05.378 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.0 2021-02-12T18:13:05.695 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.0 on osd.1 to deep-scrub 2021-02-12T18:13:05.707 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.2 2021-02-12T18:13:06.008 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.2 on osd.1 to deep-scrub 2021-02-12T18:13:06.018 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.3 2021-02-12T18:13:06.317 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.3 on osd.1 to deep-scrub 2021-02-12T18:13:06.327 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.4 2021-02-12T18:13:06.622 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.4 on osd.1 to deep-scrub 2021-02-12T18:13:06.632 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg deep-scrub 1.5 2021-02-12T18:13:06.933 INFO:teuthology.orchestra.run.smithi107.stderr:instructing pg 1.5 on osd.2 to deep-scrub 2021-02-12T18:13:06.943 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:13:26.944 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:13:27.244 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:13:27.245 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:13:27.256 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":527,"stamp":"2021-02-12T18:13:26.531764+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56661,"ondisk_log_size":56661,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":600024,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":169151,"kb_avail":188143656,"statfs":{"total":193273528320,"available":192659103744,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":173210676},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001670"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9632","reported_seq":"15196","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:13:18.565939+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:13:18.565939+0000","last_peered":"2021-02-12T18:13:18.565939+0000","last_clean":"2021-02-12T18:13:18.565939+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:13:18.565939+0000","last_undegraded":"2021-02-12T18:13:18.565939+0000","last_fullsized":"2021-02-12T18:13:18.565939+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9632,"ondisk_log_size":9632,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56661,"ondisk_log_size":56661,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738567,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":301836,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":86400,"kb_avail":94070004,"statfs":{"total":96636764160,"available":96327684096,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":88473600},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:13:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.441},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.51200000000000001,"5min":0.60499999999999998,"15min":0.67200000000000004},"last":0.42599999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.46600000000000003,"15min":0.48399999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.79300000000000004},"last":0.49199999999999999}]},{"osd":1,"last update":"Fri Feb 12 18:12:57 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46200000000000002,"15min":0.45400000000000001},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.28299999999999997},"max":{"1min":0.56200000000000006,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.45800000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.311},"max":{"1min":0.66700000000000004,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.629}]}]},{"osd":0,"up_from":0,"seq":201863463068,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084460,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298188,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":82751,"kb_avail":94073652,"statfs":{"total":96636764160,"available":96331419648,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84737076},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:13:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.432,"15min":0.44600000000000001},"min":{"1min":0.41099999999999998,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.70199999999999996},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.55000000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.36499999999999999},"max":{"1min":0.68700000000000006,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.53200000000000003}]},{"osd":2,"last update":"Fri Feb 12 18:12:52 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.27300000000000002},"max":{"1min":0.63300000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.35999999999999999},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48299999999999998,"15min":0.50800000000000001},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.755},"last":0.44}]}]}]}} 2021-02-12T18:13:27.272 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:27.274 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:27.275 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:27.275 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:27.276 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:27.276 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:27.276 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:27.277 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:27.277 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:13:47.278 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:13:47.575 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:13:47.577 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:13:47.588 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":537,"stamp":"2021-02-12T18:13:46.534158+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56736,"ondisk_log_size":56736,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604248,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":173375,"kb_avail":188139432,"statfs":{"total":193273528320,"available":192654778368,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":177536052},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001381"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9707","reported_seq":"15271","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:13:37.820991+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:13:37.820991+0000","last_peered":"2021-02-12T18:13:37.820991+0000","last_clean":"2021-02-12T18:13:37.820991+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:13:37.820991+0000","last_undegraded":"2021-02-12T18:13:37.820991+0000","last_fullsized":"2021-02-12T18:13:37.820991+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9707,"ondisk_log_size":9707,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56736,"ondisk_log_size":56736,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738571,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":305996,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":90560,"kb_avail":94065844,"statfs":{"total":96636764160,"available":96323424256,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":92733440},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:13:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.441},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.51200000000000001,"5min":0.60499999999999998,"15min":0.67200000000000004},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.46600000000000003,"15min":0.48399999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.79300000000000004},"last":0.40400000000000003}]},{"osd":1,"last update":"Fri Feb 12 18:12:57 2021","interfaces":[{"interface":"back","average":{"1min":0.46700000000000003,"5min":0.46200000000000002,"15min":0.45400000000000001},"min":{"1min":0.39100000000000001,"5min":0.36499999999999999,"15min":0.28299999999999997},"max":{"1min":0.56200000000000006,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.51700000000000002},{"interface":"front","average":{"1min":0.54000000000000004,"5min":0.58699999999999997,"15min":0.55900000000000005},"min":{"1min":0.40799999999999997,"5min":0.40799999999999997,"15min":0.311},"max":{"1min":0.66700000000000004,"5min":0.84899999999999998,"15min":0.84899999999999998},"last":0.53200000000000003}]}]},{"osd":0,"up_from":0,"seq":201863463072,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084464,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298252,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":82815,"kb_avail":94073588,"statfs":{"total":96636764160,"available":96331354112,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84802612},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:13:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.432,"15min":0.44600000000000001},"min":{"1min":0.41099999999999998,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.70199999999999996},"last":0.42099999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.55000000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.36499999999999999},"max":{"1min":0.68700000000000006,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.55900000000000005}]},{"osd":2,"last update":"Fri Feb 12 18:12:52 2021","interfaces":[{"interface":"back","average":{"1min":0.46899999999999997,"5min":0.48199999999999998,"15min":0.46600000000000003},"min":{"1min":0.374,"5min":0.34200000000000003,"15min":0.27300000000000002},"max":{"1min":0.63300000000000001,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.46999999999999997,"5min":0.48299999999999998,"15min":0.50800000000000001},"min":{"1min":0.372,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.61199999999999999,"5min":0.71299999999999997,"15min":0.755},"last":0.39600000000000002}]}]}]}} 2021-02-12T18:13:47.589 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:47.589 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:47.589 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:47.589 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:47.589 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:47.589 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:47.589 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:47.589 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:13:47.589 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:14:07.591 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:14:07.890 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:14:07.892 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:14:07.902 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":547,"stamp":"2021-02-12T18:14:06.536580+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56811,"ondisk_log_size":56811,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604440,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":173567,"kb_avail":188139240,"statfs":{"total":193273528320,"available":192654581760,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":177732660},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001477"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9782","reported_seq":"15346","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:13:57.085301+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:13:57.085301+0000","last_peered":"2021-02-12T18:13:57.085301+0000","last_clean":"2021-02-12T18:13:57.085301+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:13:57.085301+0000","last_undegraded":"2021-02-12T18:13:57.085301+0000","last_fullsized":"2021-02-12T18:13:57.085301+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9782,"ondisk_log_size":9782,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56811,"ondisk_log_size":56811,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738575,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306124,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":90688,"kb_avail":94065716,"statfs":{"total":96636764160,"available":96323293184,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":92864512},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:13:12 2021","interfaces":[{"interface":"back","average":{"1min":0.42699999999999999,"5min":0.42899999999999999,"15min":0.441},"min":{"1min":0.33300000000000002,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.51200000000000001,"5min":0.60499999999999998,"15min":0.67200000000000004},"last":0.47099999999999997},{"interface":"front","average":{"1min":0.47099999999999997,"5min":0.46600000000000003,"15min":0.48399999999999999},"min":{"1min":0.34300000000000003,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.79300000000000004},"last":0.44800000000000001}]},{"osd":1,"last update":"Fri Feb 12 18:14:00 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.28299999999999997},"max":{"1min":0.54900000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.49399999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.311},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.84899999999999998},"last":0.57799999999999996}]}]},{"osd":0,"up_from":0,"seq":201863463076,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084468,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298316,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":82879,"kb_avail":94073524,"statfs":{"total":96636764160,"available":96331288576,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84868148},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:13:17 2021","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.432,"15min":0.44600000000000001},"min":{"1min":0.41099999999999998,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.61099999999999999,"5min":0.61099999999999999,"15min":0.70199999999999996},"last":0.42899999999999999},{"interface":"front","average":{"1min":0.59799999999999998,"5min":0.58199999999999996,"15min":0.55000000000000004},"min":{"1min":0.502,"5min":0.502,"15min":0.36499999999999999},"max":{"1min":0.68700000000000006,"5min":0.78900000000000003,"15min":0.80400000000000005},"last":0.57499999999999996}]},{"osd":2,"last update":"Fri Feb 12 18:13:52 2021","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.47599999999999998,"15min":0.46899999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.27300000000000002},"max":{"1min":0.54200000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.51600000000000001},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47699999999999998,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.57499999999999996,"5min":0.71299999999999997,"15min":0.755},"last":0.495}]}]}]}} 2021-02-12T18:14:07.903 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:07.903 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:07.903 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:07.903 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:07.904 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:07.904 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:07.904 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:07.904 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:07.904 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:14:27.904 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:14:28.203 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:14:28.205 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:14:28.216 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":557,"stamp":"2021-02-12T18:14:26.539082+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56911,"ondisk_log_size":56911,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604568,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":173695,"kb_avail":188139112,"statfs":{"total":193273528320,"available":192654450688,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":177863732},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001550"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9882","reported_seq":"15446","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:14:22.757778+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:14:22.757778+0000","last_peered":"2021-02-12T18:14:22.757778+0000","last_clean":"2021-02-12T18:14:22.757778+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:14:22.757778+0000","last_undegraded":"2021-02-12T18:14:22.757778+0000","last_fullsized":"2021-02-12T18:14:22.757778+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9882,"ondisk_log_size":9882,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56911,"ondisk_log_size":56911,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738579,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306188,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":90752,"kb_avail":94065652,"statfs":{"total":96636764160,"available":96323227648,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":92930048},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:14:12 2021","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.436,"15min":0.439},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.67200000000000004},"last":0.45600000000000002},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46700000000000003,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.60599999999999998,"5min":0.61699999999999999,"15min":0.79300000000000004},"last":0.53100000000000003}]},{"osd":1,"last update":"Fri Feb 12 18:14:00 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.28299999999999997},"max":{"1min":0.54900000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48099999999999998},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.311},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.84899999999999998},"last":0.43099999999999999}]}]},{"osd":0,"up_from":0,"seq":201863463080,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084472,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298380,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":82943,"kb_avail":94073460,"statfs":{"total":96636764160,"available":96331223040,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":84933684},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:14:19 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.378,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.70199999999999996},"last":0.52500000000000002},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57799999999999996,"15min":0.55500000000000005},"min":{"1min":0.52600000000000002,"5min":0.502,"15min":0.36499999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.80400000000000005},"last":0.60099999999999998}]},{"osd":2,"last update":"Fri Feb 12 18:13:52 2021","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.47599999999999998,"15min":0.46899999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.27300000000000002},"max":{"1min":0.54200000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.66400000000000003},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47699999999999998,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.57499999999999996,"5min":0.71299999999999997,"15min":0.755},"last":0.51800000000000002}]}]}]}} 2021-02-12T18:14:28.216 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:28.217 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:28.217 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:28.217 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:28.217 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:28.217 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:28.217 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:28.218 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:28.218 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2021-02-12T18:14:48.219 DEBUG:teuthology.orchestra.run.smithi107:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2021-02-12T18:14:48.533 INFO:teuthology.orchestra.run.smithi107.stdout: 2021-02-12T18:14:48.535 INFO:teuthology.orchestra.run.smithi107.stderr:dumped all 2021-02-12T18:14:48.545 INFO:teuthology.orchestra.run.smithi107.stdout:{"pg_ready":true,"pg_map":{"version":567,"stamp":"2021-02-12T18:14:46.541593+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56986,"ondisk_log_size":56986,"up":16,"acting":16,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":16,"num_osds":2,"num_per_pool_osds":2,"num_per_pool_omap_osds":2,"kb":188743680,"kb_used":604760,"kb_used_data":430856,"kb_used_omap":0,"kb_used_meta":173887,"kb_avail":188138920,"statfs":{"total":193273528320,"available":192654254080,"internally_reserved":0,"allocated":441196544,"data_stored":440625722,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":178060340},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.001551"},"pg_stats":[{"pgid":"1.7","version":"27'6724","reported_seq":"6774","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899230+0000","last_change":"2021-02-12T18:00:07.801384+0000","last_active":"2021-02-12T18:00:34.899230+0000","last_peered":"2021-02-12T18:00:34.899230+0000","last_clean":"2021-02-12T18:00:34.899230+0000","last_became_active":"2021-02-12T18:00:07.795442+0000","last_became_peered":"2021-02-12T18:00:07.795442+0000","last_unstale":"2021-02-12T18:00:34.899230+0000","last_undegraded":"2021-02-12T18:00:34.899230+0000","last_fullsized":"2021-02-12T18:00:34.899230+0000","mapping_epoch":39,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":40,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6724,"ondisk_log_size":6724,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27541504,"num_objects":6724,"num_object_clones":0,"num_object_copies":13448,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6724,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6724,"num_write_kb":26896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":1985,"num_bytes_recovered":8130560,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.6","version":"27'6772","reported_seq":"12513","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899385+0000","last_change":"2021-02-12T17:59:44.742387+0000","last_active":"2021-02-12T18:00:34.899385+0000","last_peered":"2021-02-12T18:00:34.899385+0000","last_clean":"2021-02-12T18:00:34.899385+0000","last_became_active":"2021-02-12T17:59:44.737781+0000","last_became_peered":"2021-02-12T17:59:44.737781+0000","last_unstale":"2021-02-12T18:00:34.899385+0000","last_undegraded":"2021-02-12T18:00:34.899385+0000","last_fullsized":"2021-02-12T18:00:34.899385+0000","mapping_epoch":33,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":34,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6772,"ondisk_log_size":6772,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27738112,"num_objects":6772,"num_object_clones":0,"num_object_copies":13544,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6772,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6772,"num_write_kb":27088,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4928,"num_bytes_recovered":20185088,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.1","version":"27'6816","reported_seq":"16942","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900236+0000","last_change":"2021-02-12T18:00:26.881505+0000","last_active":"2021-02-12T18:00:34.900236+0000","last_peered":"2021-02-12T18:00:34.900236+0000","last_clean":"2021-02-12T18:00:34.900236+0000","last_became_active":"2021-02-12T18:00:26.875844+0000","last_became_peered":"2021-02-12T18:00:26.875844+0000","last_unstale":"2021-02-12T18:00:34.900236+0000","last_undegraded":"2021-02-12T18:00:34.900236+0000","last_fullsized":"2021-02-12T18:00:34.900236+0000","mapping_epoch":43,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":44,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6816,"ondisk_log_size":6816,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27918336,"num_objects":6816,"num_object_clones":0,"num_object_copies":13632,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6816,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6816,"num_write_kb":27264,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":5023,"num_bytes_recovered":20574208,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"1.0","version":"47'9957","reported_seq":"15521","reported_epoch":"47","state":"active+clean+scrubbing+deep","last_fresh":"2021-02-12T18:14:42.023697+0000","last_change":"2021-02-12T18:01:00.045810+0000","last_active":"2021-02-12T18:14:42.023697+0000","last_peered":"2021-02-12T18:14:42.023697+0000","last_clean":"2021-02-12T18:14:42.023697+0000","last_became_active":"2021-02-12T17:59:52.758905+0000","last_became_peered":"2021-02-12T17:59:52.758905+0000","last_unstale":"2021-02-12T18:14:42.023697+0000","last_undegraded":"2021-02-12T18:14:42.023697+0000","last_fullsized":"2021-02-12T18:14:42.023697+0000","mapping_epoch":35,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":36,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":9957,"ondisk_log_size":9957,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27578368,"num_objects":6733,"num_object_clones":0,"num_object_copies":13466,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6733,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6733,"num_write_kb":26932,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4848,"num_bytes_recovered":19857408,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.2","version":"27'6533","reported_seq":"12150","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.910128+0000","last_change":"2021-02-12T18:00:34.908986+0000","last_active":"2021-02-12T18:00:34.910128+0000","last_peered":"2021-02-12T18:00:34.910128+0000","last_clean":"2021-02-12T18:00:34.910128+0000","last_became_active":"2021-02-12T18:00:34.902790+0000","last_became_peered":"2021-02-12T18:00:34.902790+0000","last_unstale":"2021-02-12T18:00:34.910128+0000","last_undegraded":"2021-02-12T18:00:34.910128+0000","last_fullsized":"2021-02-12T18:00:34.910128+0000","mapping_epoch":45,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":46,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6533,"ondisk_log_size":6533,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":26759168,"num_objects":6533,"num_object_clones":0,"num_object_copies":13066,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6533,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6533,"num_write_kb":26132,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4755,"num_bytes_recovered":19476480,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.3","version":"27'6687","reported_seq":"6736","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899237+0000","last_change":"2021-02-12T18:00:16.843341+0000","last_active":"2021-02-12T18:00:34.899237+0000","last_peered":"2021-02-12T18:00:34.899237+0000","last_clean":"2021-02-12T18:00:34.899237+0000","last_became_active":"2021-02-12T18:00:16.837531+0000","last_became_peered":"2021-02-12T18:00:16.837531+0000","last_unstale":"2021-02-12T18:00:34.899237+0000","last_undegraded":"2021-02-12T18:00:34.899237+0000","last_fullsized":"2021-02-12T18:00:34.899237+0000","mapping_epoch":41,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":42,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6687,"ondisk_log_size":6687,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27389952,"num_objects":6687,"num_object_clones":0,"num_object_copies":13374,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6687,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6687,"num_write_kb":26748,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":2039,"num_bytes_recovered":8351744,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.4","version":"27'6688","reported_seq":"12423","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.899278+0000","last_change":"2021-02-12T17:59:36.712322+0000","last_active":"2021-02-12T18:00:34.899278+0000","last_peered":"2021-02-12T18:00:34.899278+0000","last_clean":"2021-02-12T18:00:34.899278+0000","last_became_active":"2021-02-12T17:59:36.707680+0000","last_became_peered":"2021-02-12T17:59:36.707680+0000","last_unstale":"2021-02-12T18:00:34.899278+0000","last_undegraded":"2021-02-12T18:00:34.899278+0000","last_fullsized":"2021-02-12T18:00:34.899278+0000","mapping_epoch":31,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":32,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6688,"ondisk_log_size":6688,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27394048,"num_objects":6688,"num_object_clones":0,"num_object_copies":13376,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6688,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6688,"num_write_kb":26752,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4902,"num_bytes_recovered":20078592,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.5","version":"27'6809","reported_seq":"16844","reported_epoch":"46","state":"active+clean","last_fresh":"2021-02-12T18:00:34.900589+0000","last_change":"2021-02-12T18:00:00.783894+0000","last_active":"2021-02-12T18:00:34.900589+0000","last_peered":"2021-02-12T18:00:34.900589+0000","last_clean":"2021-02-12T18:00:34.900589+0000","last_became_active":"2021-02-12T18:00:00.777631+0000","last_became_peered":"2021-02-12T18:00:00.777631+0000","last_unstale":"2021-02-12T18:00:34.900589+0000","last_undegraded":"2021-02-12T18:00:34.900589+0000","last_fullsized":"2021-02-12T18:00:34.900589+0000","mapping_epoch":37,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":38,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2021-02-12T17:57:23.470569+0000","last_clean_scrub_stamp":"2021-02-12T17:57:23.470569+0000","log_size":6809,"ondisk_log_size":6809,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":27881496,"num_objects":6808,"num_object_clones":0,"num_object_copies":13616,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":6808,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":6809,"num_write_kb":27230,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":4975,"num_bytes_recovered":20369456,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":8,"stat_sum":{"num_bytes":220200984,"num_objects":53761,"num_object_clones":0,"num_object_copies":107522,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":53761,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":53762,"num_write_kb":215042,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":33455,"num_bytes_recovered":137023536,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":440410112,"data_stored":440401968,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":56986,"ondisk_log_size":56986,"up":16,"acting":16,"num_store_stats":3}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738583,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":306252,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":90816,"kb_avail":94065588,"statfs":{"total":96636764160,"available":96323162112,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":92995584},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:14:12 2021","interfaces":[{"interface":"back","average":{"1min":0.44700000000000001,"5min":0.436,"15min":0.439},"min":{"1min":0.374,"5min":0.33300000000000002,"15min":0.33300000000000002},"max":{"1min":0.62,"5min":0.62,"15min":0.67200000000000004},"last":0.504},{"interface":"front","average":{"1min":0.47599999999999998,"5min":0.46700000000000003,"15min":0.47999999999999998},"min":{"1min":0.35599999999999998,"5min":0.34300000000000003,"15min":0.34300000000000003},"max":{"1min":0.60599999999999998,"5min":0.61699999999999999,"15min":0.79300000000000004},"last":0.45500000000000002}]},{"osd":1,"last update":"Fri Feb 12 18:14:00 2021","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46400000000000002,"15min":0.46000000000000002},"min":{"1min":0.35799999999999998,"5min":0.35799999999999998,"15min":0.28299999999999997},"max":{"1min":0.54900000000000004,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48699999999999999},{"interface":"front","average":{"1min":0.53900000000000003,"5min":0.57499999999999996,"15min":0.56799999999999995},"min":{"1min":0.36899999999999999,"5min":0.36899999999999999,"15min":0.311},"max":{"1min":0.80800000000000005,"5min":0.80800000000000005,"15min":0.84899999999999998},"last":0.56499999999999995}]}]},{"osd":0,"up_from":0,"seq":201863463085,"num_pgs":0,"num_osds":0,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":0,"kb_used":0,"kb_used_data":0,"kb_used_omap":0,"kb_used_meta":0,"kb_avail":0,"statfs":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},{"osd":1,"up_from":28,"seq":120259084476,"num_pgs":8,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":298508,"kb_used_data":215428,"kb_used_omap":0,"kb_used_meta":83071,"kb_avail":94073332,"statfs":{"total":96636764160,"available":96331091968,"internally_reserved":0,"allocated":220598272,"data_stored":220312861,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":972,"internal_metadata":85064756},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Fri Feb 12 18:14:19 2021","interfaces":[{"interface":"back","average":{"1min":0.46800000000000003,"5min":0.438,"15min":0.44600000000000001},"min":{"1min":0.378,"5min":0.34200000000000003,"15min":0.32600000000000001},"max":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.70199999999999996},"last":0.34999999999999998},{"interface":"front","average":{"1min":0.58699999999999997,"5min":0.57799999999999996,"15min":0.55500000000000005},"min":{"1min":0.52600000000000002,"5min":0.502,"15min":0.36499999999999999},"max":{"1min":0.69999999999999996,"5min":0.69999999999999996,"15min":0.80400000000000005},"last":0.52200000000000002}]},{"osd":2,"last update":"Fri Feb 12 18:13:52 2021","interfaces":[{"interface":"back","average":{"1min":0.44600000000000001,"5min":0.47599999999999998,"15min":0.46899999999999997},"min":{"1min":0.33600000000000002,"5min":0.33600000000000002,"15min":0.27300000000000002},"max":{"1min":0.54200000000000004,"5min":0.91600000000000004,"15min":0.91600000000000004},"last":0.436},{"interface":"front","average":{"1min":0.47899999999999998,"5min":0.47699999999999998,"15min":0.50600000000000001},"min":{"1min":0.39600000000000002,"5min":0.36599999999999999,"15min":0.36399999999999999},"max":{"1min":0.57499999999999996,"5min":0.71299999999999997,"15min":0.755},"last":0.85199999999999998}]}]}]}} 2021-02-12T18:14:48.546 INFO:tasks.ceph:pgid 1.7 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:48.547 INFO:tasks.ceph:pgid 1.6 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:48.547 INFO:tasks.ceph:pgid 1.1 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:48.547 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:48.547 INFO:tasks.ceph:pgid 1.2 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:48.547 INFO:tasks.ceph:pgid 1.3 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:48.547 INFO:tasks.ceph:pgid 1.4 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:48.547 INFO:tasks.ceph:pgid 1.5 last_scrub_stamp 2021-02-12T17:57:23.470569+0000 time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=17, tm_min=57, tm_sec=23, tm_wday=4, tm_yday=43, tm_isdst=-1) <= time.struct_time(tm_year=2021, tm_mon=2, tm_mday=12, tm_hour=18, tm_min=0, tm_sec=55, tm_wday=4, tm_yday=43, tm_isdst=0) 2021-02-12T18:14:48.547 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa/tasks/ceph.py", line 1892, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa/tasks/ceph.py", line 1280, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2021-02-12T18:14:48.560 INFO:teuthology.misc:Shutting down mds daemons... 2021-02-12T18:14:48.560 INFO:teuthology.misc:Shutting down osd daemons... 2021-02-12T18:14:48.561 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2021-02-12T18:14:48.561 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T18:14:48.589 INFO:tasks.ceph.osd.0:Stopped 2021-02-12T18:14:48.590 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2021-02-12T18:14:48.590 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T18:14:48.670 INFO:tasks.ceph.osd.1:Stopped 2021-02-12T18:14:48.670 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2021-02-12T18:14:48.670 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T18:14:48.763 INFO:tasks.ceph.osd.2:Stopped 2021-02-12T18:14:48.763 INFO:teuthology.misc:Shutting down mgr daemons... 2021-02-12T18:14:48.763 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2021-02-12T18:14:48.763 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T18:14:48.781 INFO:tasks.ceph.mgr.x:Stopped 2021-02-12T18:14:48.781 INFO:teuthology.misc:Shutting down mon daemons... 2021-02-12T18:14:48.782 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2021-02-12T18:14:48.782 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T18:14:48.832 INFO:tasks.ceph.mon.a:Stopped 2021-02-12T18:14:48.833 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2021-02-12T18:14:48.833 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T18:14:48.884 INFO:tasks.ceph.mon.b:Stopped 2021-02-12T18:14:48.884 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2021-02-12T18:14:48.884 INFO:teuthology.orchestra.run:waiting for 300 2021-02-12T18:14:48.935 INFO:tasks.ceph.mon.c:Stopped 2021-02-12T18:14:48.935 INFO:tasks.ceph:Checking cluster log for badness... 2021-02-12T18:14:48.936 DEBUG:teuthology.orchestra.run.smithi107:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v 'but it is still running' | egrep -v 'overall HEALTH_' | egrep -v '\(OSDMAP_FLAGS\)' | egrep -v '\(OSD_' | egrep -v '\(PG_' | egrep -v '\(OBJECT_' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2021-02-12T18:14:49.002 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T18:14:49.002 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2021-02-12T18:14:49.460 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T18:14:49.460 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2021-02-12T18:14:49.557 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T18:14:49.557 DEBUG:teuthology.orchestra.run.smithi107:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2021-02-12T18:14:49.620 INFO:tasks.ceph:Archiving mon data... 2021-02-12T18:14:49.621 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860/data/mon.a.tgz 2021-02-12T18:14:49.621 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2021-02-12T18:14:49.636 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.MAuZEvCDvx 2021-02-12T18:14:49.636 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.MAuZEvCDvx 2021-02-12T18:14:49.898 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.MAuZEvCDvx 2021-02-12T18:14:49.979 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.MAuZEvCDvx is 703KB 2021-02-12T18:14:50.079 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.MAuZEvCDvx 2021-02-12T18:14:50.092 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860/data/mon.b.tgz 2021-02-12T18:14:50.092 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2021-02-12T18:14:50.145 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.vSplhOwAI7 2021-02-12T18:14:50.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.vSplhOwAI7 2021-02-12T18:14:50.394 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.vSplhOwAI7 2021-02-12T18:14:50.474 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.vSplhOwAI7 is 729KB 2021-02-12T18:14:50.573 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.vSplhOwAI7 2021-02-12T18:14:50.587 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860/data/mon.c.tgz 2021-02-12T18:14:50.587 DEBUG:teuthology.orchestra.run.smithi107:> mktemp 2021-02-12T18:14:50.640 INFO:teuthology.orchestra.run.smithi107.stdout:/tmp/tmp.6qhMYz6D1B 2021-02-12T18:14:50.640 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.6qhMYz6D1B 2021-02-12T18:14:50.888 DEBUG:teuthology.orchestra.run.smithi107:> sudo chmod 0666 /tmp/tmp.6qhMYz6D1B 2021-02-12T18:14:50.969 DEBUG:teuthology.orchestra.remote:smithi107:/tmp/tmp.6qhMYz6D1B is 729KB 2021-02-12T18:14:51.068 DEBUG:teuthology.orchestra.run.smithi107:> rm -fr /tmp/tmp.6qhMYz6D1B 2021-02-12T18:14:51.081 INFO:tasks.ceph:Cleaning ceph cluster... 2021-02-12T18:14:51.082 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2021-02-12T18:14:51.145 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2021-02-12T18:14:51.145 DEBUG:teuthology.orchestra.run.smithi107:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2021-02-12T18:14:51.209 INFO:teuthology.orchestra.run.smithi107.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2021-02-12T18:14:51.215 INFO:tasks.ceph:Archiving crash dumps... 2021-02-12T18:14:51.216 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/lib/ceph/crash to /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860/remote/smithi107/crash 2021-02-12T18:14:51.217 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2021-02-12T18:14:51.280 INFO:tasks.ceph:Compressing logs... 2021-02-12T18:14:51.280 DEBUG:teuthology.orchestra.run.smithi107:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-12T18:17:37.654 INFO:tasks.ceph:Archiving logs... 2021-02-12T18:17:37.723 DEBUG:teuthology.misc:Transferring archived files from smithi107:/var/log/ceph to /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860/remote/smithi107/log 2021-02-12T18:17:37.724 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /var/log/ceph -- . 2021-02-12T18:18:21.075 ERROR:teuthology.run_tasks:Manager failed: ceph Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa/tasks/ceph.py", line 1892, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa/tasks/ceph.py", line 1280, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2021-02-12T18:18:21.090 DEBUG:teuthology.run_tasks:Unwinding manager install 2021-02-12T18:18:21.098 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/task/install/__init__.py", line 619, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/run_tasks.py", line 176, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa/tasks/ceph.py", line 1892, in task osd_scrub_pgs(ctx, config) File "/home/teuthworker/src/git.ceph.com_ceph-c_5a3430b93fcfcd31d38534758b8997c894688c4b/qa/tasks/ceph.py", line 1280, in osd_scrub_pgs raise RuntimeError('Exiting scrub checking -- not all pgs scrubbed.') RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2021-02-12T18:18:21.102 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2021-02-12T18:18:21.102 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2021-02-12T18:18:21.145 DEBUG:teuthology.run_tasks:Unwinding manager clock 2021-02-12T18:18:21.155 INFO:teuthology.task.clock:Checking final clock skew... 2021-02-12T18:18:21.155 DEBUG:teuthology.orchestra.run.smithi107:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2021-02-12T18:18:21.179 INFO:teuthology.orchestra.run.smithi107.stderr:bash: ntpq: command not found 2021-02-12T18:18:21.184 INFO:teuthology.orchestra.run.smithi107.stdout:210 Number of sources = 4 2021-02-12T18:18:21.184 INFO:teuthology.orchestra.run.smithi107.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2021-02-12T18:18:21.184 INFO:teuthology.orchestra.run.smithi107.stdout:=============================================================================== 2021-02-12T18:18:21.184 INFO:teuthology.orchestra.run.smithi107.stdout:^- hv01.front.sepia.ceph.com 3 6 377 56 -4672us[-4672us] +/- 128ms 2021-02-12T18:18:21.184 INFO:teuthology.orchestra.run.smithi107.stdout:^* hv02.front.sepia.ceph.com 2 6 377 61 +1605us[+1609us] +/- 37ms 2021-02-12T18:18:21.184 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 61 -3524us[-3524us] +/- 93ms 2021-02-12T18:18:21.185 INFO:teuthology.orchestra.run.smithi107.stdout:^+ hv04.front.sepia.ceph.com 3 6 377 58 -581us[ -581us] +/- 64ms 2021-02-12T18:18:21.185 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2021-02-12T18:18:21.197 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2021-02-12T18:18:21.197 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2021-02-12T18:18:21.357 DEBUG:teuthology.orchestra.run.smithi107:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2021-02-12T18:18:21.417 DEBUG:teuthology.orchestra.run.smithi107:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av '\(comm="dmidecode"\|chronyd.service\|name="cephtest"\|scontext=system_u:system_r:nrpe_t:s0\|scontext=system_u:system_r:pcp_pmlogger_t\|scontext=system_u:system_r:pcp_pmcd_t:s0\|comm="rhsmd"\|scontext=system_u:system_r:syslogd_t:s0\|tcontext=system_u:system_r:nrpe_t:s0\|comm="updatedb"\|comm="smartd"\|comm="rhsmcertd-worke"\|comm="setroubleshootd"\|comm="rpm"\|tcontext=system_u:object_r:container_runtime_exec_t:s0\|comm="ksmtuned"\|scontext=system_u:system_r:logrotate_t:s0\)' 2021-02-12T18:18:21.444 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-12T18:18:21.444 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2021-02-12T18:18:21.465 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2021-02-12T18:18:21.506 INFO:teuthology.task.internal:Duration was 1718.928244 seconds 2021-02-12T18:18:21.507 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2021-02-12T18:18:21.528 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2021-02-12T18:18:21.529 DEBUG:teuthology.orchestra.run.smithi107:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2021-02-12T18:18:21.573 INFO:teuthology.orchestra.run.smithi107.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2021-02-12T18:18:21.820 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2021-02-12T18:18:21.820 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi107.front.sepia.ceph.com 2021-02-12T18:18:21.820 DEBUG:teuthology.orchestra.run.smithi107:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/*.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2021-02-12T18:18:21.878 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2021-02-12T18:18:21.879 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2021-02-12T18:18:21.954 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2021-02-12T18:18:21.966 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2021-02-12T18:18:21.967 DEBUG:teuthology.orchestra.run.smithi107:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2021-02-12T18:18:22.019 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2021-02-12T18:18:22.029 DEBUG:teuthology.orchestra.run.smithi107:> sudo sysctl -w kernel.core_pattern=core && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2021-02-12T18:18:22.083 INFO:teuthology.orchestra.run.smithi107.stdout:kernel.core_pattern = core 2021-02-12T18:18:22.084 DEBUG:teuthology.orchestra.run.smithi107:> test -e /home/ubuntu/cephtest/archive/coredump 2021-02-12T18:18:22.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2021-02-12T18:18:22.138 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2021-02-12T18:18:22.149 INFO:teuthology.task.internal:Transferring archived files... 2021-02-12T18:18:22.151 DEBUG:teuthology.misc:Transferring archived files from smithi107:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/sseshasa-2021-02-12_17:37:51-rados-wip-dmclock-fix-high-client-profile-distro-basic-smithi/5877860/remote/smithi107 2021-02-12T18:18:22.151 DEBUG:teuthology.orchestra.run.smithi107:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2021-02-12T18:18:22.215 INFO:teuthology.task.internal:Removing archive directory... 2021-02-12T18:18:22.216 DEBUG:teuthology.orchestra.run.smithi107:> rm -rf -- /home/ubuntu/cephtest/archive 2021-02-12T18:18:22.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2021-02-12T18:18:22.290 INFO:teuthology.task.internal:Not uploading archives. 2021-02-12T18:18:22.291 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2021-02-12T18:18:22.301 INFO:teuthology.task.internal:Tidying up after the test... 2021-02-12T18:18:22.302 DEBUG:teuthology.orchestra.run.smithi107:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2021-02-12T18:18:22.317 INFO:teuthology.orchestra.run.smithi107.stdout: 262189 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Feb 12 18:18 /home/ubuntu/cephtest 2021-02-12T18:18:22.318 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2021-02-12T18:18:22.340 DEBUG:teuthology.run_tasks:Exception was not quenched, exiting: RuntimeError: Exiting scrub checking -- not all pgs scrubbed. 2021-02-12T18:18:22.340 INFO:teuthology.nuke:Checking targets against current locks 2021-02-12T18:18:22.439 CRITICAL:teuthology:Uncaught exception Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/run.py", line 409, in main run_tasks(tasks=config['tasks'], ctx=fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/run_tasks.py", line 204, in run_tasks raise SystemExit(1) SystemExit: 1 During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/virtualenv/bin/teuthology", line 33, in sys.exit(load_entry_point('teuthology', 'console_scripts', 'teuthology')()) File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/scripts/run.py", line 38, in main teuthology.run.main(args) File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/run.py", line 412, in main report_outcome(config, archive, fake_ctx.summary, fake_ctx) File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/run.py", line 269, in report_outcome nuke(fake_ctx, fake_ctx.lock) File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/nuke/__init__.py", line 266, in nuke for unnuked in p: File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/parallel.py", line 98, in __next__ resurrect_traceback(result) File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/parallel.py", line 30, in resurrect_traceback raise exc.exc_info[1] File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/parallel.py", line 23, in capture_traceback return func(*args, **kwargs) File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/nuke/__init__.py", line 280, in nuke_one keep_logs = ctx.keep_logs File "/home/teuthworker/src/git.ceph.com_git_teuthology_87d40f79d7da6caf6da06c190938773bf745140c/teuthology/config.py", line 255, in __getattr__ raise AttributeError(name) AttributeError: keep_logs